Hi
HFSS works in the frequency domain which implies a linear regime. So my answer to the linear/nonlinear question is no: HFSS can not be used for the simulation of nonlinear electromagnetic phenomena (like those involved in nonlinear optics). For these kind of phenomena the prevalent method is FDTD (Finite Difference Time Domain) but I have seen somebody using TLM also (Transmission Line Method).
Concerning the initial question (Simulation of optical filters on HFSS) in principle the answer is positive because optical filters are still linear components. In a real problem, as pointed out by alionxj, the utilization of HFSS could be very problematic (or not applicable at all) when the problem involves complex materials such as highly dispersive media (which at some frequency can be characterized by negative dielectric constants) and real conductors (which at optical frequency have to be considered as very lossy and dispersive dielectrics).