Dear,
I have a rule of thumb that I always use in microwave filter design: never expect more than 45 dB rejection on a stopband of a real filter. If you need more than this value use two filters with lower order cascaded with some sort of buffer in between. For 5 to 6 GHz you can use gain blocks like NBB 400 or NBB 500 between the cells.
I developed two mathcad files to analyze suspended striplines single and coupled. The files are attached. Using these files I did some good filters up to 2.2GHz. My attempt in 5GHz range was really bad, but I think it is worthy you try it.