What I am saying is that it looks like in your open loop gain equation, you are multiplying by N. You instead want to be dividing by N. This is possibly the source of your problem.
N is the divider ratio. If you are dividing by 10, for instance, then any phase deviation will be divided by 10 also, and you will only get 1/10 of the voltage out of the phase detector. Even for just a N=10, that will give you a 40 dB error (20 +20 dB going from multiplying to dividing by N=10).