Looking at your connector's data sheet, I suspect that they teaked the layout for that microstrip (!) feed with some magic compensation to achieve good wideband match. For your GCPW, things might be different.
Simulation:
I first checked ADS linecalc results, and noticed that you designed for the quality control (cavity resonator) permittivity value er=3.48 instead of the recommended "circuit design" value er=3.66. This doesn't explain the differences that you observe, but I wanted to bring it to your attention anyway.
For simulation, I used the thickness that you specified, but looking at the RO4350B data sheet it doesn't seem to be a standard RO4350B thickness value !?
I created a model in Empire XPU (3D FDTD EM) using CPW ports, and added vias from top ground to bottom ground.
To make room for the extra feedlength of the semi-calibrated CPW ports, I extended the line and via fence. The line length between the reference planes ("P1E" and "P2E") is approx. the same as your total PCB length.
Below are results for both er=3.48 and er=3.66. Return loss is much better than what you observe, with resonable agreement to the 48 Ohm impedance from linecalc.
Conclusion: I really think you have a discontinuity at the coax - PCB interface that causes poor return loss. In simulation, such a discontinuity can be caused by incorrect port definition. In measurement, I don't trust your coax transition. When I was working in the microwave lab, we used much better (more well defined) transitions at these frequencies.
Given the same line length, simulation and measurement then would show a similar mismatch pattern, for different reasons.
- - - Updated - - -
In the meantime, I also created a true 3D model including the SMA, to show what can go wrong. The edge mount SMA has a small airgap (200µm) on the sides and is connected to the PCB using solder (red boxes).
With this, I get rather similar results to your measurements. Note that the line itself was ok, simulation in my previous post. So issue is caused by the connector and how it is connected to ground.
Note the narrow gap between SMA and PCB, and note the solder in the ground path.