You should be more clear in your terms. Not that there are rigid definitions for terms like 'HOT', but you should be careful to be clear.
HOT would typically mean a mains circuit with a significant potential to ground, or perhaps in some areas isolated from ground.
NEUTRAL would mean a mains circuit current-carrying conductor with a potential near ground. Think of it as a return to ground, because that is how it is often wired.
Depending on your situation, one or both of your incoming mains wires could both be HOT.
ISOLATED means isolated and there is no hot or neutral unless you ground one side, either accidentally or deliberately. If I understand you correctly, you are using a normally connected scope to measure the outputs of an isolation transformer that is not otherwise connected--is that right? In that case, your single probe measurement will only get you noise, not a reliable reading. It could be anything, but you'll likely see your mains frequency, probably distorted at some voltage that I can't easily guess. The mains GFCI will not trip, unless perhaps you use a 1X probe with a 50R scope input and your isolation transformer leaks very badly or is under water.
Two probes and MATH is the same principal as a differential probe and will work fine, only limited by how well your scope does the math. However, the point (one of them anyway) of the isolation transformer is that you now can somewhat arbitrarily select your ground point and measure from there. So if you simply grounded one side of your isolated output, you can now measure the other side directly. Of course, this defeats the touch safety feature of isolation. You can't have both at the same time.