Welcome to our site! EDAboard.com is an international Electronics Discussion Forum focused on EDA software, circuits, schematics, books, theory, papers, asic, pld, 8051, DSP, Network, RF, Analog Design, PCB, Service Manuals... and a whole lot more! To participate you need to register. Registration is free. Click here to register now.
Have you looked at the loop gain transfer function of your system? If you have a low phase margin, you will get high-Q resonant poles near the crossover frequency. You can also see the effect of this as overshoot and ringing in the time-domain waveforms. Create a step change in one of your input variables and see if the output is overshooting and ringing. You should examine the poles and zeros (as paulux said) of the loop gain transfer function and make sure you have at least 45° of phase margin. More phase margin is better. Good luck with your design.