regarding the use of an emitter follower...
The output impedance of an emitter follower is 1/gm, so the series resistor you need at the output is 50 -1/gm. Keep in mind that you introduce a voltage divider at the output which will lower your gain, thus with 50 in series you end up with ~-6dB instead of your desired 2-3dB.
If you use common emitter circuit (without degeneration) the gain is gm * RC. The intrinsic rce of the device is high, so you can neglect it if you keep RC low. So you could choose a small RC (50Ohm) and then adjust the gm by adjusting the current Ic.
If you use emitter degeneration your gain is (first order) Rc/Re and you can control your with a lot less dependence on the current.
Matching for 50Ohms is usually done for max. power transfer, keep in mind that the back on the envelope calculations described here are focused on voltage gain and not power gain!
Lastly, each circuit type has different noise and linearity trade offs. In your case I assume you don't care about noise, but more about linearity. If you run a source follower at low currents, linearity will be really bad.
Before you make a decision I would suggest you determine what the max. signal level is that you want to pass thru that circuit.