It's not verifiable, that the balun design suggested in your literature is actually operational as such in the lower frequency range. Measuring two baluns back-to-back proves nothing, unfortunately. You have to use a single balun and correct differential (and additional common mode) termination to check the real characteristics with a VNA.
In my opinion, it's obvious, that a balun for e.g. 100 MHz must either have large dimensions or use a magnetic core (a transformer). I'm sure, that a printed balun without a core isn't feasible for the intended frequency range.