If you used a compensated x10 'scope probe, you could have easily measured the output of the function generator without any attenuation or distortion.
In the case or your experiment involving serial transmission. I suspect it's mains pick-up which is the problem, rather than transmission line reflections as 100Hz is too low for that to be a problem. The wires you used weren't shielded, so picked up noise from the mains which was removed by connecting 50 Ohm to 0V as the source impedance of the mains hum would be quite high (many Mega Ohms), yet the source impedance of the serial signal would be very low (under 50 Ohms). Try using shielded cables and this should improve, without the need to add a resistor.