Okay~ let's try this.
you normalise the output data: when it is larger than 0.9V, give it the value +1; if it is smaller than 0.9V, give it the value -1, then it has two levels +1 and -1, instead of 1.8 and 0, do the fft and plot it again to see what happens.
I am doing a DSM now, I also meet this problem before, my output has 1.2V and 0V, if I directly do fft on it, I cannot get the dB I expect; after I normalise it, it looks very good~
I dunt know how you do the fft, if you deal with the output data already as I write above, then it might be an other story...