You hook a spectrum analyzer to the IF output port, and drive the LO and RF inputs with a synthesizer.
You set up Frf to be the desired signal and look at the IF out power level in dBm.
You then set up Frf to be the image, but the same distance from the LO as the desired signal. You look at the IF out power level.
The difference in the 2 IF out power levels is your image rejection.