1a. Assume that TI applications examples for a synthetic balun (discrete phase-shift 2*C and 2*L) are correct and use same values assumed that your PCB layout also is very similar.
1b. Buy a balun, connect it to VNA and do your impedance matching. Alternatively buy or build your own
balun and calibrate the VNA for 200 Ohm and measure to find out best component values for a synthetic balun to convert down to 50 Ohm unbalanced. Coaxial balun is cheap and quick but you must know its limitations. It will only show correct values for one frequency and measured impedance, as showed on VNA, must be recalulated, if VNA is set for 50 Ohm.
If calculating a synthetic balun, I assume 86.5+j43 is not including any PCB load or trace length.
2. You must measure radiated harmonics for total circuit, including antenna. As an example, if less good values are selected for the synthetic balun, can it cause to much harmonics (personal experience with CC1101). A spectrum analyzer and a reference antenna and a measurement chamber, all calibrated according to actual standard, are needed to do a qualified job, but a bench test with a spectrum analyzer and any antenna can show if there are obvious problems with radiated harmonics.
CC1101 is in my opinion a bit sensitive and can cause a lot of harmonics if not correct matched and well decoupled at Vcc. Have never worked with CC1110 but guess it is similar in RF design.