jtag optocoupler
JTAG is a digital level interface only. There are no analog or intermediate level signalling. The typical non-isolated JTAG interface has pull-up resistors on most of the lines. If the PC side is driving the outputs as open collector, then you need to add pullup resistors to the PC side of TCK, TDI, and TMS.
The output signal from the microcontroller board to the PC is TDO. This also needs a pull-up resistor as most TDO outputs are open collector.
While I am not familiar with your microcontroller, I do know that some other units can disable the JTAG with fuse bits within the microcontroller to allow the JTAG to be used for other purposes. Are you sure your microcontroller is JTAG enabled?
Are you sure there are no wiring errors? Scope out the TCK pin of the microcontroller it should be toggling during the JTAG operation. TDI should also be toggling, but at a slower rate since it is data. TMS is the select line between data input words and control input words. TMS will often be low for long periods but should go high a few times during the cycle. TDO should be checked back at the PC side. It too should toggle.
What is the PC side? Are you using the parallel port? If so, then be aware that you sometimes have to change the parallel port configuration in BIOS to get it working. Also, the USB to parallel port adapters do NOT work for this sort of bit-banging. Believe me, I tried!.