This will happen if you're trying to simulate an electrically large model. A few possible solutions:
-If you're doing a frequency sweep, look at a lower top end - change your range so you're not simulating as high of frequencies
-If possible, use PEC instead of finite conductors - depends on your model
-If doing a frequency sweep, set to discrete rather than fast sweep - it will take forever but HFSS often simulates at one frequency correctly, it just can't handle the fast sweep
-If possible tweak your solution parameters - what's your delta-S or delta-E, how many converged passes do you require, how much does the mesh refine each pass, etc
3G RAM is not a lot for HFSS. You may just be out of luck depending on your model.