Using a voltage divider to 50 Ohms and coax feed to Spectrum Analyzer on analog preamp out may help isolate noise by viewing spectral density. Examine noise above f/2 sampling rate also.
- - - Updated - - -
I recall designing something with 12 bit ADC using Burr Brown Hybrid in 70's and had no difficulty getting the noise input below LSB but I had some serious problems with dead spots at intermediate logic transition levels like xxxxxx001111 to 010000. Since it was inside the chip, I surmised it was due to logic level shifting the analog ground Vref inside. I changed the chip from an Xray inspected 883B type to Industrial Type and the problem mysteriously went away. This lack of monoticity can be a common issue with logic currents sharing analog ground. , but unrelated to noise. FWIW.
If you have a 12 bit DAC, my best test was to use scope and take Output-Input vs Input in XY mode and discover the lack of monoticity and modulation of noise due to hysteresis effects of positive feedback to Vref from logic state current.
I suggest you do the same to discover all sources of error.
If Vref is noisy, or A_gnd , you can get similar noise problems from lack of filtering here.
YOu can also inject noise externally by making assumptions where you think noise is getting in to see if it gets worse, then fix it.
Tony