Re: squar or sine wave ?
From a noise point of view, it is definitely better to mix with a square waave LO. This reduces the flicker noise dramatically ( you can refer to the classic paper on noise in mixers by Hooman Darabi; it's titled "Noise in RF CMOS mixers; a simple physical model" )
I think the effect on gain will be minor. Because even when you use a sine wave, the actual mixing process will be approximately a multiplication with a square wave ( assuming a small switching transient )
Effect on IIP3 is rather not clear however. You'd better find out yourself by simulation