The reference books for mode matching technique and rectangular iris waveguide filters are:
Field Theory of Guided Waves, R.E. Collin
Waveguide Handbook, N. Marcuvitz
Microwave Filters, Impedance-Matching Networks, and Coupling Structures, G. Matthaei
You may find something 'useful' at the following link www.ebookee.info. You will understand why
Anyway, I designed waveguide filters in the past and I have some notes and demonstrations, but in Italian!!!. I should have some HFSS file as well. Unfortunately I don't have much time for translating or scanning my notes but I can provide you with an HFSS example project, if you want.