Hi,
you are getting closer.
Since PSK results in symmetric (doubled) bandwith than the input you can theoretically only use 25MHz of analog input signal bandwidth.
But now you have square wave with it´s harmonics....
You need to suppress all input frequencies above 25MHz.
A square wave has no 2nd harmonic, only odd harmonics. Therefore if you want to go to the third harmonic, then there is only 8Mbaud possible.
I´m no expert in data modulation, but I think the margin is too small.
Maybe there is a member that is experienced with PSK modulation ... he can give you more "realistic" informations.
If your input frequency is not properly bandwidth limited, then you will intrude in the next channel.
Klaus