Source/emitter/cathode followers all suffer from th same effect:- on positive going signals the device "switches on" and its output impedance is ~ i/gm || Rs/Re/Rk. On negative going signals, the device will try to switch off due to stray capacity holding the source positive while the gate goes negative, hence the output impedance will be Rs/re/Rk. This would account for the assymetry of the wave form. Exactly what is happening around the zero volts is another matter. Is your Vdd clean?
Frank