I'm studying an IEEE paper("http://ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=272096&url=http%3A%2F%2Fieeexplore.ieee.org%2Fxpls%2Fabs_all.jsp%3Farnumber%3D272096").
Can anybody let me know any published text books or documents to explain how to determine the compensation capacitor size in Class AB output stage?
Depends on your load capacitance (which determines - together with the output impedance - the dominant pole). As a rule of thumb, use 30..35% of the output capacitance for the compensation capacitor(s).
Then with this load capacitance and your largest to be expected load resistance check the (open loop) Bode diagram for your phase margin. Correct the compensation caps if necessary.