As it works with a "1" input, but not with a "0" input I can see your frustration!
My instant thought was contact bounce, but that is not affected by the data in state.
At the moment my best suggestion is put some decoupling capacitors close the IC on the supply rail, in case switching is generating supply glitches and hence this symptom. Try 100nF.
If that doesn't help try reducing the Cap on the clock line. It is just possible that if the clock rise and fall times are too slow that this could cause some odd effects.
If debounce of the switch is an issue look in to some SR latchs made with cross coupled gates on the switch.