I think the best sources for your question are going to be from Microstrip Filters for RF / Microwave Applications by Hong and Lancaster or Microwave Filters for Communication Systems by Cameron. Both of these methods describe synthesis of filters using the coupling matrix, which is a very handy tool for making filters using coupled cavities. Changing the size of the iris will change the coupling between the cavities, thus affecting filter characteristics. I'm not sure what the differences are in circular and rectangular irises, I suspect that's mainly a manufacturing concern.