The kT/C noise is just there independent of your input amplitude. So if you want to sample something really small then yeah you need to handle large caps unless there's something else you can change in the architecture.
There are two ways around it that I can think of. First is using an amplifier with low noise so that the signal is larger. If you're handling really small signals, you may want to have an amplifier anyway because at 100MHz rate, you'd be loading your input quite a bit so it's not a huge loss to gain it up a bit. Second way is addressing the issue with other techniques. There is this paper that I remember from some time ago, but to be honest it's been a while since I read it so I don't remember it well. It may still provide a starting point.
https://ieeexplore.ieee.org/document/6814935 The main point is, there are some techniques that can be used conditionally.