Why does BER in Rayleigh Fading Channel for 64-QAM or 256-QAM remain constant after some Eb/N0 (let's say 25 dB) and will not decrease although we increase the Eb/N0
Is there any method to compensate this effect (except Equalization or Channel Estimation)?
You have a mistake. BER in Rayleigh Fading does not remain constant for 64, 256 or M-QAM with increasing Eb/N0. However, for such constellations, the slope of the error curve can not be obsevered in low SNR. Just try to increase SNR. With increasing SNRs (maybe 30,40dB) even the BER of 256-QAM can decrease to 1e-5 s.