You need to do the following steps:
1) determine the transmit and receive requirements for your system as a black box
2) Choose appropriate components for the Tx and Rx components
3) Plug the component performance values into a cascade analysis program. This will compute, for example, the Rx total noise figure from you stated LNA NF and gain, mixer NF and loss, IF amp NF and gain, etc. It will compute cascaded gain, NF, IP3, etc.
4) If the system performance is poor, look over the cascade analysis spread sheet, replace the worst component or two with a better one, and run the analysis again.
5) After you are close, try varying things, like the receiver gain due to AGC circuits, etc.
Good luck. It is 1/3 art, 2/3 calculations.