I presume you are talking about an analog video 75 ohm coax connection? It is usually expecting DC coupling, even with line-by-line level clamping there must be no DC drop during a line interval. So which balun type with sufficient low corner frequency are you imagining?
Analog video over CAT5 will normally use DC capable differential amplifiers and possibly equalizers to reduce the waveform distortion caused by the cable.
Frequency dependent attenuation of CAT5/6/7 cable will be considerably higher than of good (sufficient diameter) 75 ohm video cable. So if your main concern is video quality, you'll opt for coax cable.
Digital transmission will promise better quality anyway.