it could be a number of things.
I would hook up a voltmeter to the VCO control line, and see if the DC voltage is nearing a supply rail....if say you have a 3.3 V supply rail, and you need 3.0 volts to lock up the VCO...maybe the op amp or charge pump can not put out that much voltage, and it falls out of lock.
I would also suspect that the PLL is unstable. You need phase margin and gain margin to have a stable PLL control loop. Often times this means adding a control loop "zero" to the feedback circuit. Otherwise you have two poles (the VCO is one pole, and the integrator op amp a 2nd pole) and you have an inherently unstable loop without adding a zero. Another common term for this is to add a "lead-Lag network".
You might have higher order poles, like due to shunt RF bypass capacitors in the tuning part of the VCO, that further degrade the PLL stability, and need to be compensated. This is a common problem if you are using a really big control loop bandwidth...like 1 MHz wide.
You might have not taken into account "Transport Lag" in your control loop design. If you have a big divisor ratio N....what happens is some digital counter counts N RF input pulses, and when N is achieved, it puts out ONE digital pulse. So if N=600,345.....it takes T = (600345) * (the VCO period) to get an output from the divider. THAT can be a significant time delay. A big time delay has the same effect as having poor phase margin in your control loop design.
HOW TO TEST for stability? I like to add a small digital phase shifter to the clock port, and then look at the phase response of the locked VCO. If you step the clock 2 degrees in phase, and the VCO has a nice damped phase change that settles out quickly...you probably have a good control loop circuit. If you put your clock phase step into the system, and the VCO phase goes crazy with all sorts of ringing and instability...you have a marginally stable, or even unstable, PLL design.
HOW TO TEST if you are actually phase locked? Sometimes you THINK it is phase locked, but you are kidding yourself. Move the clock frequency a few Hz. If the VCO moves N x a few Hz....it is locked. It needs to be EXACTLY N times! (N being the divisor, or fractional divisor ratio you think you are programming in)