Hi,
You can not do AC simulation to check for phase margin using cadence, just like u do for other circuits. For PLL, u write transfer function (TF) for each block, say
for PFD - 1/2pi
for VCO - Kvco/s (integrator)
Divider - 1/N
for LPF - 1/sC/R+1/sc etc etc.
Write down open loop TF - G(s) for the full system. Compare with the standard 2nd order or 3rd order TF. This depends on whether u have used 1st order or 2nd order loop filter. Get the equation for damping factor and ωn.
Then write closed TF - G(s)/1+G(s).H(s).
Do Matlab (or Octave) simulations for open-loop and closed loop TFs. From open-loop TF, for the required damping factor, set the value of R nad C of loop filter. Put these values in closed loop TF and findout BW and phase margin.
Only after doing this, u have to proceed for transient simulations. Thats the correct way, as far as i know.