Think in terms of signal to noise ratio. At both the input and the output of a passive network like a filter the noise floor is the same, derived from K*T*B. However, any signal would be attenuated by the loss of the filter. So the signal to noise ratio at the input would be higher that the signal to noise ratio at the output. That difference in S/N ratio, the loss in the filter, is the noise figure.
For a active circuit the noise is not the same at the input and the output because of the noise added by the active circuit. The noise figure though is still the difference in the S/N ratio input vs output.