Actually, HFSS is operating correctly. Your assumptions are not correct. When the mode is below cutoff, ie evanescent, then all of the energy going into the model decays evanescently before it reaches the pec cap at the end. Therefore, there is no energy that can reflect and exit back through the port, so the Return Loss (energy reflected back into the port) is near 0 % (or a very small - dB value). However, once the propagating mode is allowed, the energy is entirely reflected from the pec end and therefore all the energy is reflected back into the port, ie Return Loss should be 0 dB as the reflection power is 100 % (no loss as pec is implied).
Also, as this is a symmetric waveguide, there will be degenerate modes that need to be accounted for. Please see attached modified project.
Have Fun!