After all is said and done, the PSS simulations turn out to be optimistic by about 7-10 dBC/Hz.
Pls. read this paper as a reference.
Jacobsson, H., et al, “Low-Phase-Noise Low-Power IC VCOs for 5-8-GHz Wireless
Applications”, IEEE, Trans. MTT, Vol. 48, No. 12, December 2000, pp. 2533-2539
It may be a setup problem ----> goto spectre manual
It may be a topology problem ----> goto your professor/boss
It may be a shorter simulation than is necessary ----> Kill a weekend
Try not to include the effects of the transient startup.
Try some initial conditions.
Try using an ideal mixer on chip and setup up a phase noise simulation as you would in the lab to get a handle on what is going on since you mentioned that the transient simulation is working. (kill several weekends)
Use ADS to simulate phase noise, if the circuit is not too too big. Again you will be off by 3-5 dBc/Hz (says the paper above)
Hope it helps.