Is it better to switch to your secondary instead of reloading your primary?

I usually switch to my secondary if I'm in a gunfight and I'm out of ammo, but my friend calls me stupid for doing it. Is it bad to do this?