size of HFSS model
1. Try installing 3 GB of RAM. HFSS v10 is compiled to utilize a PC with 3 GB of RAM, although there will be an upper limit. The other solution is to use a 64-bit OS such as Linux or Unix. This circumvents the entire memory addressing limitation of a 32-bit OS.
2. Radiation boundaries can be placed up to lambda/4 away from the antenna. PMLs can be placed even closer, but the penalty is that the PML itself must be meshed since it is a volume instead of a surface boundary condition. One wavelength away is too big.
3. For planar antennas such as p_atches, you should consider using a planar EM tool such as IE3D or Designer. The solve times will be much faster since MoM codes only need to mesh the surfaces and not the entire volume. I wouldn't recommend using a fields-based, 3D method (FEM) when a source-based, 2.5D method (MoM) will suffice.
Good luck.