Re: FracN PLL phase noise
The behaviout of your PLL seems quite 'normal'. It is pretty common that the inband phase noise is worsened when you enable the Fractional part (ie. using an input ≠ 0).
The inband noise is a combination of many differents contribution within the fractional pll.
One is from the fact that the sampling instance in the phase-detector is not a constant. That is, the time from your charge pump is enabled once until the next time it is enabled is changing from clock to clock, so you will have an unequal sampling which downconverts quantisation noise down to your inband frequencies.
Another one is from the non-linearities in the different blocks (phase detector/charge pump etc) which also cause the phase noise to fold down to baseband.
So you do have a lot of contributors, and you will probably be able to decrease them to some level by narrowing the bandwidth. But in an ideal world you had to prevent them where they come from. eg. design some more linear components (good luck with this..) and for the non uniform sampling some proposes a S/H switch in the output of the chargepump (but take care of the spurs this might introduce!).
Regarding your noise isolation then it is always nice to have steep edges on your phase detector input, but I guess that even though you share the signal then you probably have some buffers inside your phase detector. So putting in an inverter will probably just load your signal just as much as the sigma delta modulator does.
Well maybe I misunderstand the intention of the inverter... If you want to place the inverter to make some time difference between the sigma delta activity and to the time when the charge pump is active, then I have done a lot of experiments with that with no significant difference.
Finally regarding your chosen loop bandwidth I must say that it seems a bit high! If you want an idea of where the quantization noise breaks through your inband noise floor then Perrots nice program can give you a good indication
https://www-mtl.mit.edu/researchgroups/perrottgroup/tools.html#plldesign.
Practical loopbandwiths are usually much lower. Both due to the quantization noise but also due to inband spurs. This is the reason that many have tried to make some 'bandwidth extension' to ease direct modulation.