I have made some comparison measurement between (VCO / VCXO) based PLL synthesizers. I got a bit contradicting result. At the same operating frequency 400 MHz, the VCXO based synthesizer has much better close-in phase noise (as expected) BUT the phase noise analyzer claims that the VCXO based has much higher residual FM compared to the VCO based PLL synthesizer. The RMS jitter is better for the VCXO based PLL synthesizer. As I have understood, the RMS jitter relates to residual FM. At least you can find such integral equations describing how to calculate the residual FM from the measured phase noise curve.
Could this be related on the fact that the VCXO output is ~square wave and it has some duty-cycle variation 45/55? The VCO has sinewave output with duty cycle very close to 50%.
Take a look near to the carrier of your's VCXO PLL output. You may have low frequency perturbations
or small jumps in the order of 10 - 20Hz.
It may happen very easy - because of the small VCXO gain the loop may become a litle bit instable,
especilally if the division ratio is low ( reference frequency is high). It can be also due to cycle slipping in the phase detector.
Karote, Make sure that you have the right amount of FM for the VCXO other wise if you have to much you may want to consider increasing the voltage for the timing filter VCXO Manufacturers, Designers, and Suppliers of Products let me know if any of the vcxo listed here will help with the frequency.