Let's say, you have: a sensor, an anti-aliasing filter, an A/D.
Theoretically (Nyqist theorem) the cutoff frequency of the anti-aliasing filter should be 1/2 of the sampling frequency. In practice the cutoff frequency should be even smaller; it depends (among other things) on the number of poles in the filter.
The sampling frequency is related to the mesage signal frequency: fs>=2fm
where fs is the sampling frequency and fm is the high frequency component of the message signal ( information signal)
Cut off freq is the frequency at which the power reduces to half of it's maximum value
I am proposing you to have cut off frequency for the filter before the SH or ADC according to:
Fcutoff <= Tsampling/4
or even
Fcutoff <= Tsampling/8
Please note that
Fsampling = 4*Fss
guaranteed the band width of the sampler not worse than 3 dB.
You should take into consideration the characteristic of the filter before SH (if any) and the ADC when you calculate the bandwidth of the entire sampling and conversion system.
I think that is a good idea to test your system with sinusoidal signal (SS) and to apply the paper