Noise Figure is the added noise ratio from the device you are using.
Conversion gain is an actual loss in passive mixers due to(mostly) diode mixing the signals. When in an active mixer, you use an active device to eliminate the loss in the mixer and u get an actual gain instead of a say -7db conversion gain.
Noise figure on the other hand, is a result of the active pieces you are using in the mixer. Your transistor has a dc operating point, which consumes current; hence adds noise signal on top of your desired signal. Only way to decrease is to implement the mixer with a low-current low-noise transistor which may be specified for LNA. That way, you can decrease some, but it will always add some because of the active devices.