Of course each iteration uses more ram.. it is using ever more tetrahedra to mesh.
Waveguide filtters are tricky, as they are large compared to lambda but there are some get-arounds:
If it is a normal type of filter, these are very well understood, and there are standard formulae for calculating a starting point.
They are usually symmetrical in more than one axis: you can use Symmetry planes to drastically reduce the size of your model (and speed up the solve time!)
You need to of course set the impedance multiplier correctly. There are relatively good documents how to do this in HFSS, and previously (version 8 days) was very standard, as CPU power was a fraction of todays.
Use the "assign mesh operation" to manually mesh critical areas before you start.
Using the optimiser in HFSS is rather optimistic. Much better is to use some other way to design the waveguide filter, and then test the results in HFSS and make final adjustments if necessary. A week of optimisation is quite a long time
If you really want to make your own fancy type of waveguide filter
There are far better ways to do this.. for example using mode matching software (see Mician uWave Wizard) which can optimise you a near perfect solution in minutes.
You can download an eveluation version, which I am sure will get you extremely close to the final result far faster than HFSS.
>after say 100 iterations HFSS waits for a really long time after each iteration >displaying messages like 'updating all derived data centrally'.
I am not surprised... it is shovelling around a LOT of data.
Hope my comments are helpful.