Hey mtwieg!
The noise power in the unwanted [lower, in this case] sideband (and any signals that happen to be there) are added in to the final output spectrum, thereby raising the noise floor by the 3 dB you observed. Unfortunately, fiddling the shape of the passband (assuming it's optimally matched to the signals of interest and no wider than strictly necessary) or LO frequencies doesn't help either - all that does is move the origin of the fold.
So in my case, I only need a baseband filter with a fc of 50KHz in order to recover all the information in the 100KHz BW narrowband signal, correct? Assuming I have quadrature detection, of course. In that case, is there any penalty to my SNR? I should have 100KHz worth of signal, and 100KHz worth of noise, same as the original narrowband signal, correct?
There *is* a solution though, and it's alluded to by my note at the end about the real and (possible) complex representations of the mixer output. If complex downconversion (also called quadrature mixing) is performed instead, you can arrange for the unwanted sideband to be intrinsically 'canceled out' in the mixing operation and avoid the problem you're observing. While the cost is a more complicated mixing stage than just a single mixer, for a fixed (and relatively low) IF frequency like yours it can be implemented accurately without much difficulty. You can find references and details on the web by searching for "image rejecting (or rejection) mixers". The "phasing method of SSB generation" also uses the same techniques, so you might find something useful there too.
Fortunately I've already got I/Q mixers in the hardware, along with baseband filters on each. I expected I'd need them at some point, but I'm not sure what to do with them. In fact, my narrowband signal is pretty much entirely real, so when I set the LO phase correctly I actually will only get signal on the quadrature channel, and nothing but noise on the I channel. So in this case what do I do with the information from the I channel? You mentioned that IQ can be used to "cancel out" the negative frequencies, but I don't understand what you mean there. Your pictures are perfectly clear, but how does IQ demodulation solve the issue with frequency "folding"?
I am no DSP expert, but I think if you just run the adc at >20 MHz clock speed (2 x the nyquist rate) you can separate negative frequencies from positive frequencies. yes/no?
The noise figure is not 3 dB higher, per se, than with a low IF frequency. It is just that you have both the negative and positive frequencies shown in 0 to 5 MHz baseband.
Unfortunately, there's no way I can sample that fast. Best I can do at the moment is 625Ksps (simultaneous samples on the I and Q channels).
I also considered digital IF downconversion as described here:
**broken link removed**
But this requires a good antialiasing BPF at the center frequency, which is inconvenient for me since my fc will likely change arbitrarily.