I don't know the range of your output frequency, but its clearly that there are spurs. But the SFDR should be same or better than in the datasheet.
The voltage compiliance of the output mean, that a the IOUT and the /IOUT Pin of the device is able to swing around +/- 0,5V from AVVD.
So if you connect two resistors from AVVD to this pins, and you have an reference current of 10mA these resistors are limited to a value of R<=50Ohm.
If you use an center tapped transformer with ratio 1:1 you have to terminate the secondary side with equal or less then 100Ohm.
To calculate the best filter you can use the tool **broken link removed**. It works fine. I use it to start my simulations of the passive output section. So you are able to find the best filter.
I know its the hardest part to get the best results. I spent most of my time with spectral measurements.