Where is your "hold" capacitor?
Without it, all you'll see is toggling and the switching charge on
turnoff (absent from your ideal switch model) will determine the
(alleged) "hold" voltage.
Hard full scale gate drive will give you a common mode dependent
sample pedestal voltage. Imbalance in Cdg (and if high Zin, Cgs) will
give you a charge offset too.
Put a layout-realistic hold capacitor on the output and resimulate.
Look at close-in port currents at the sample-to-hold edge for
charge imbalance and resulting pedestal.