Some model railways use an AC signal to both power and transmit data to the locomotive. It's rectified in the locomotive to power the motor and the changes in duty cycle or frequency are used to tell it the speed and direction of travel.
I've never worked with such a system, so wouldn't know how to 'scope it. I would start by checking which conductor is connected to mains ground and connect the 'scope ground pin to that. If neither conductor is earthed, then I suppose it doesn't matter which side the 'scope ground pin is connected to.