hard to say without simulation.
this asymmetry leads to a conclusion that the only part of the circuit that is not symmetrical causes the problem - the output current mirror for single-ended output. One possibility is that when your Vcm is high, biasing current for the input differential pair becomes higher (if the output resistance of the current source transistors small), so the more critical side of the output current mirror (the one that is not the output) cannot provide such a current with voltage headroom allowed. In that case, one should use bigger transistors in output, and better current source at input.
Once more, it's hard to say more without simulation.