To add one more:
The ref and div input signals to the phase/frequency detector are not always 50% duty. So if the falling edge comes near the rising edge of the other input it could also introduce nonlinearity.
To add another one:
More for characterization. If both input edges are aligned, so zero phase difference, the delay or minimum on time could be a little longer than the minimum pulse if the phase delay is significant.
Until I forget:
Test for symmetry. If phase difference behavior is not truly symmetric all sigma-delta calculations will fail.
Please stop here: (until I spend too much time)