gain range for LVDS RX
Well, you need to produce a full scale logic output from a
spec-minimum input overdrive. That sets a DC minimum
of something like (VOH-VOL)/(VIH-VIL). But you need more
than this for good switching delay.
The problem might be more like a lack of hysteresis,
or ground bounce greater than hysteresis along with a weak
high frequency CMRR.
You might want to supplement the DC hysteresis (if any) with
a time-domain hysteresis that will inbibit reswitching for some
brief time (less than half a period of your fastest data rate).
Inspect the input waveform to be sure it is not showing a
mid-transition plateau (as might come from a grossly
misterminated line) or spending too long getting through
the linear input band. If it spends significant time between
VIL and VIH, that's undefined input hence undefined output.
You can only do so much with bogus input, and wouldn't
want to go redesigning a part for a board / line problem.