In order to find out the presence of modulation you can use numerous techniques.
First of all, try to estimate instantaneous frequency (IF) of the signal. It may be done with the use of the conception of analytic signals and Hilbert transform. But there are several limitations on the signal you must take into account. These limitations are connected with narrow-band spectrum, otherwise the Hilbert transform is deprived of any sense (remember rectangular impulse, which has wide-band Fourier spectrum - it's amplitude envelope, counted by Hilbert transfor, is useless!!!). For counting instantaneous frequency use the techniques of numerical differentiation, and Richardson extrapolation.
Besides, IF can be also estimated by wavelet transform
Another way - depict time-frequency distributions (Wigner-Wille, wavelet, Hilbert). The view of spectrum helps find out the presence of amplitude and frequency modulations. The disatvantage - it can't be estimated in numbers from the view.
Finally, have you heard about EMD (Empirical mode decomposition)? It's one of the most powerful contemporary techniques of adaptive signal processing. Besides, it provides you with the opportunity to count IF. I have been working with EMD for 2 years already, therefore i have got enormous experience. Also I have Matlab code on EMD. Indicate your e-mail and i"ll send them to you.
With respect,
Dmitrij