but If you are referring to how to demonstrate that a certain propagation mode has a certain cut-off frequency you have just to excite this mode and simulate it on the frequency range of interest.
You can you the prorogation constant, which is the im(gamma(portNumber:mode)). If you could be more specific as to what you are trying to optimize about the waveguide it would be helpful.