Many references are all come to an equation which is difficult to solve. And many references mention the design briefly or only use software to simulate. Anyone can design the waveguide filter step by step just like the parallel coupling microstrip filter design?
Ansoft corp. have one good tutorial on waveguid filter design.I remember someone have upload this tutorial on forum. It's easy to copy its method to your design basing on how much your understand of your design.
I'm sorry I also can't find it.Hope this presentation can help you"Rapid_Elliptical_Filter_Design.pdf",you can download from Ansoft website(workshop archives ).