RMS jitter requires first that you collect many samples,
to do the math upon.
Samples worth having, must include all of the internal
and external jitter inducing stimuli. When talking fS,
every parasitic inductor matters (all the way from the
ground plane point of reference, to the chip core).
Big fun there and slow, slow solution times.
Then if it were me I'd be making a veriloga "widget"
that catches the delay from every transition of the
reference clock, to the signal of interest, spit them
to a file and grind it out in Excel later. If I wanted a
numerical result.
On the other hand I usually don't care that much
about more than a few significant digits, and will
just eye-plot the waveform raster-overlaid on
itself, measure the left and right edges of that
smear, and call it P-P jitter (somebody else can
do the transform if they can say what the real world
distribution shape is - normal, bimodal (DJ), etc.).