Quite general challenge to all current available solvers. High frequency need denser meshes, higher order basis functions, in general more # of unknowns if your problems domain size remains the same. So for high frequency results, only way is to increase mesh, and apply high order basis. Depends on your problem size, this problem can easily eat up 10Gb memory even nowadays.
Basically, (pure Nyquist theory) you need to have at least two spatial sampling points per-wavelength, this means at 20GHz (1.5cm wavelength in freespace), your cell size is at most 0.75cm. if you use substrate whose Er is 2, 4, 9 or whatever, cell size must be smaller.
High order basis function can improve cell size, but its self is expensive as well.
FDTD is matrix free and can handle much denser meshes. However, its dispersion error sometimes is really killing people and in order to obtain the same accuracy with FEM, a much denser grid than FEM elements must be applied, which turns out about the same cost to get the same accuracy.
So you question is actually a big challenge, which academic is trying to solve for years.