The RC time constant comes into play here.
Your input resistance is probably high. Perhaps it goes low when the input is shut off, then it goes high when the input is turned on? And the capacitor must acquire it's charge all over again, through a very high resistance?
Or, perhaps you switch in the input, causing the capacitor to pull the bias very high, or very low?... Until it acquires a charge, then the amplifier can again operate in its normal region.
Anyway the problem might be solved if you reduce the capacitor value.