the reference signal is ideal
because i simulate the it's transient reponse, the vco's nonideal isn't present
i think that the main cause is vco's control voltage it's ripple is large
as below
i adjust the gain of vco, and change the bandwidth of loop filter
but the ripple of vco's control voltage is present yet
maybe the mismatch of charge pump's source & sink currents results, the spurs?
i will try it
That does not look like "phase jitter". That looks a lot like your control loop is ringing (and almost completely unstable) at around a 10 KHz open loop bandwidth.
Try changing R2 fro 293 to 2200 ohms and resimulate and let us know if the "jitter" goes away.
With your existing design, you have a control loop zero at 33 KHz, which is not doing anything to help the stability problem down at 10 KHz.