rbw vs vbw
The basics are:
Set the frequency bandwidth to see the area you want to. If you are looking at close in phase noise, sweep +/- 100 KHz for instnace.
Leave the video bandwidth on auto, unless you see a lot of jagged data that you think might not be real, then lower the bandwidth a little. Narrow video bandwidth throws away a lot of information, so if you want an hones measurement leave it wider.
RBW depends on where you want to measure. If you care about phase noise at 1 Mhz offset from the carrier, set it at 10 KHz. If you want a good measurement at 100KHz offset, set it at 1 KHz. If you want a measurement at 2 KHz from the carrier, set it as small as you can go, like 10 Hz.
RBW is the only thing that grossly effects the measurement. Lets say at 100 KHz offset, you measure -50 dBc phase noise with RBW set to 1 KHz. So you are letting in a 1 KHz width of noise in the spectrum analyzer's IF. 1000 Hz is 30 dB higher than 1 Hz. So the phase noise of the signal is -50 dBc -30 dBc/Hz = -80 dBc/Hz.
Got that? That is pretty much it.
If you are really picky, you can add or substract a few dB from the final phase noise number to correct for IF filter shape--read your spectrum analyzer manual.