Make sure your calculations are all correct. Cutoff of waveguide, size of waveguide and coax, impedance match between coax and waveguide, and correct port renormalization for your system's impedance you are using.
From the picture do you have an outer conductor for your coax? Almost looks like you just have inner conductor and dielectric. And then the face of the dielectric would be your waveports. I'm guessing if your S11 is incorrect it is showing around 0 to 5 dB return loss and probably not a good match to the waveguide.