If you want to transmit 6b/s/hz, you have to use 64 QAM.
modulate signal using simple qammod(signal, M)
generate channel model using rayleighchan provide multi paths
use y=filter(chan,signal) to apply rayleigh channel
use awgn(y,snr) to add up noise
demodulate usign qamdemod(sign, M);
plot ber v/s snr using biterr(x,y).
vary snr from 0 to 30;
observe results.
observe scatterplot of modulated signal, filter signal, noise signal which will help in equalization algorithm.