Hy,
Well, i have looked your shematic. It seem to me that, on your branchline, you have just put the four lambda/4 lines without optimize your system.It is evident also that the simulation give you a higher frequency. In fact, when you connect your four lines it appears that this line are less long than your lambda/4 calculated for 3 GHz. So you must have to introduce 1 variables like L1+epsilon for a first couple of line at Zc1 (L1 fixed = lambda/4 at 3 GHz , epsilon a very small variable) and L2+epsilon for the second couple of line at Zc2 (L2 fixed = lambda/4 at 3 GHz, epsilon the same very small variable). Indeed, optimize epsilon in order to have dB(S12)=3 dB or phase(S12/S13)=90°.
Plus i suggest you to add four acess lines to your branchline coupler.
Good simulations.