I'd bet that the 'scope says your 100mV with respect
to ADC input reference carries millivolts of noise. If you
can resolve that, even. How many codes (LSBs) would
that be?
You might experiment with things like bond-wire shorting
the ADC input to the ADC reference (or some on-chip
known-not-noisy signal source) and see what comes
out the back when nothing even touches the outside
world.
Because the outside world is grimy and lumpy.
This is why much ADC ATE testing has come to be done
by large sample statistics, the mean or median is the
best you can do. A one-shot test will almost never give
you the same answer, twice. And there goes your drift,
for reliability testing.