You need better amplitude match and more accurate quadrature. What you have is not bad for a first attempt.
Local oscillator rejection relative to your wanted output will depend on signal level and mixer imbalance. Injecting DC into the IF port of the mixer can null the leakage. Getting it all right is an iterative process of adjusting LO leakage, signal phase and signal ampilitude. The narrower the bandwidth the easier it will be.
Search for IQ mixer sideband rejection, that should get you plenty of application note hits.