Yes, you can do it. Will the mixer performance be "optimized"? Maybe not.
Specifically, the port to port leakage and the spurious signal levels (M x N products) may be different if you drive the IF port with the high power instead of the LO port. Sometimes the only way to tell is to actually test a unit.
Similarly, for some applications, you may choose to drive the RF port with the high poiwer, and use the LO port as a low power input or output.
About the only mixers you can not do this with are newer "comutating types", where you are driving fet gates instead of diode quads.