The KLI4104 looks like a nice chip. I have experience integrating a Sony ILX554B with Xilinx Spartan 3 board:
Have you decided on your analog front end for correlated double sampling and digitization? What aspects of the timing specs are you having trouble with? There is a bit of a learning curve to understanding this stuff.