trade offs are:
1) input impedance match
2) bandwidth
3) large signal performance (P1dB, IP3, IP2)
1) the optimum NF is achieved at impedances other than 50 or 75 Ohm...-> google for noise matching
2) matching for lowest possible noise will (in many cases) be narrow band
3) in order to operate the device for best noise performance requires a bias point that is non optimal for large signal behavior of your amp.
I provided a very basic, simplified explanation. The trade offs depend on the type of device (MOSFET, BJT) and the type of circuits you choose (common gate, common source etc).
You could study a few text books that derive the equations for each case to get a deeper understanding.