Guess it is a problem with electrical delay. Check electrical delay is correct set:
Measure S11 open, PCB connected but no components. Repeat for S11 short, with aid of a short wire connected to ground.
If electrical delay is correct set should Smith chart show a single dot and almost no reactive load. If not, manually adjust delay and check again.
If it not is possible to get reasonable well defined dots, check that cables and connectors not are faulty. For example can SMA connectors by worn out, center pin bent..
Also check that calibration kit seems correct by visually inspection, compare with connecting a SMA open if calibration kit parameters are correct set in VNA.
If electrical delay seems correct adjusted, then verify that only minor reactive load can be seen for different resistor values when measuring S11 and S21 (and later more interesting, verify calibration by measuring reactive components)
If possible I use predefined calibrations standards, but also have my own calibration standards to be able to do in circuit calibration.
For example requires some measurement that a very thin, flexible and lossy coaxial cable is used when measuring inside cellphones as there not is space enough to route a thicker cable.
Other reasons to do in circuit calibration is when parts of in phone existing PCB traces must be included in calibration. Maybe not a big problem at 600 MHz and short traces but at 2.4GHz can correct calibration be the difference between if a inductor or capacitor should be added to a network.
Have no experience from Advantest and guess it can differ a lot between different brands of VNA, but setting up my own VNA calibration parameters is not so complicated if not too high precision is required.
My precision requirement is in the range that I want an error less then one E12 step, to be able to select correct component values for filters and impedance matching network.
What do I use for the standard definition tables?
Can only tell my procedure, which can be different for another typ of VNA:
First must you know what it is you want to define, which probably is depending on what types of job you want to do.
In may case I often solder a semi rigid 100 mm long cable in an existing PCB with intention to make an embedded antenna, starting from the point where the cable is soldered.. Other end of this cable have a SMA connecto, connect to a low loss cable and VNA. My first need of definition point was therefore the end of this coaxial cable and upper frequency limit 3 GHz.
I have done five exactly similar cables loaded to shield with:
1. short
2. open
3. 4*200 Ohm parallel 0402
4. 4.7 nH Murata 0402 15HS
5. 4.7 pF Murata 0402
Set VNA to span 2-3 GHz
Select defining a new cal.standard and sett all offsets and reactive values to "0".
Make a SOL calibration with this setup and the cables 1-3.
Set electrical delay with cable 1 and 2. No error offset should seen between cable 1 and 2.
Measure cable 4 and adjust Open->C0 for best correct reading compared with from by manufacturer given S11. (I have done a simple software that does this step automatically.)
Adjust C1 for best "tilt"
Verify 50 Ohm load and if needed adjust Load->C0.
Repeat and if needed fine adjust remaining parameters.
A similar coaxial cable is then soldered in a PCB, and electrical delay adjusted by a knife at PCB and it now gives a relative correct reading.
This is a very homebrewed setup with several limitations. Use at your own risk.