Using math for differential measurements will work, but you need to make sure that both signals, Ch1 and Ch2 in your case, are within the dynamic range of the ADC of the scope. If one channel or the other has the output of the ADC pinned to a positive or negative value, you're going to get some strange looking waveforms with the resultant Ch1-Ch2.
Some scopes warn about over-range conditions and some don't. To be sure you're not over-range, you should make sure the signal on each channel is fully on the vertical area of the screen, and then do the math for Ch1-Ch2.