Well, DC offset is one fundamental limit, but since the latch is allowed only a limited time to resolve the input, you should also consider the minimum input amplitude that will cause it to settle to the full swing within a limited time.
Latch structure has infinite DC gain.But Latch has large random offset.As a example,we will use preamplifier for the latch and its random offset can be cancelled by the preamp.You should arrange the settling time seperately for the preamp and the latch to get the smallest settling time.You can find some solutions from a paper by P.R.Gray.But I cannot recall its name clearly.