I don't see a datasheet for ts1620a-20 under your link. I also notice that the device is offered by different manufacturers, ts1620 seems to be a kind of collective term.
I believe the question has been answered as far as we can. It's obviously a 5V device, expected to work with typically up to 5V polarisation voltage (measured between Vdd and V0). I found a typical polarisation voltage of 5V mentioned in a ts1620-1 datasheet, means the wiper would be typically near the GND pot end.
Did you check the actual polarisation voltage with a meter? According to maximum ratings, V0 can be supplied with a negative voltage (up to 11V below VDD), if the contrast can be improved with a polarisation voltage above 5V, you have probably got a device with rather marginal specification. Or it's simply defective.