Gregoost,
I think your question is about using larger number of constellation points in modulation. As mentioned by deepuj, higher order constellations require more snr for maintaining the same bit error rate as that obtained in binary signalling schemes. You could have binary signalling in PSK also, where the symbols are anti-podal ie, the 0 and 1 are 180 degrees out of phase with each other. This is Binary Phase Shift Keying. If we increase the number of constellations to 4, we get QPSK, but for the same BER, we would now require double the SNR (+3dB). Here the symbols would be 90 degrees apart.
In my experience, in very noisy channles such as HF radio, higher order constellations are difficult to demodulate. Further more, due to the poor group delay response characteristics of HF radio, adjacent symbols 'spread' and interfere with each other. In such cases, antipodal signalling gives the best results, even though the bit rate is low.
I guess it depends on the channel you intend to use the modem on. If the SNR is very good, you must go for higher order constellations, because it would improve the data transfer rate. If it is too noisy and non-linear, lower order constellations would be the only option.
Regards.