It depends on the modulation you're using. Typically you can start from a simplified measurement model based on the AM/AM and AM/PM curves of the amplifier. For example if you know that you loose 0.5 dB for the bigger amplitude from the constellation, you compensate that by driving the amplifier with 0,5 dB extra. Same goes for phase distortion.
For some modulation formats both AM/AM and AM/PM are important (eg 64-QAM), for other only phase distortion is important (GMSK e.g.).