Ground noise is probably worse, because ground is often a
reference for "single ended" signals that are really processed
differentially, so subject to gain rather than designed
attenuation (as supplies see, PSRR). The problem is worse
when you are not dealing with "a" ground, but a ground
network with various series and shunt elements and noise
injected at different points (i.e. reality).
Differential (explicitly) signals remove much of the ground
sensitivity leaving only CMRR / PSRR attenuated ground
"input".