First and the oldest technique is to do all the calculations by hand using already-made formulas that you can find in the filter books.
Second is to use a linear RF simulator that is using linear models (as transmission lines, coupled lines, etc)
Third is to use an EM simulator, making your own models.