subtract power spectral densities
Hi,
Are you doing this in simulation? is your noise white? Because if your noise is white (like randn function in MATLAB) then you cannot eliminate the noise by subtraction. Subtraction for white noise is like addition and just adds to the power of noise.
Does filtering the out of band noise work for you? Then you can subtract the filtered version of your signal from your original signal and what is left is the noise with almost same level of power in the out-of band and more power (almost double) in the in-band.
Did I understand your question properly?
Regards - TS