while trying to make minimal differences of 2 signals visible via the math channel, I found some behaviour I can not explain.
[...]
Hi Peter,
with both input channels acquired at 100 mV/div and the math channel displayed at 500 µV/div, the math trace is effectively showing the difference between both channels at about 200x vertical magnification relative to the acquisition scale.
This is somewhat similar to taking a single-shot acquisition at 100 mV/div, stopping the scope, and then zooming it vertically to 500 µV/div. The trace may look very clean at 100 mV/div, but after that much vertical magnification the remaining acquisition noise becomes quite obvious.
Also keep in mind that subtracting two channels does not subtract their uncorrelated noise. Instead, the noise from both channels contributes to the math result.
The Magnova has a very low-noise front end, but at this amount of vertical magnification even the remaining noise becomes clearly visible. Using HiRes and/or averaging, as you already mentioned, can reduce the visible noise significantly (the acquired 12-bit ADC data is processed and stored with 16-bit resolution).
If the goal is to measure a very small difference between two relatively large signals, a suitable low-noise differential probe may also be worth considering. This would allow the differential component to be presented directly to the ADC at a more suitable input range, instead of relying on such a large vertical magnification of the subtraction result.
Regarding the different DVM AC RMS readings: you noted about 530 µVrms at 100 mV/div (800 mV full screen) versus 70 µVrms at 10 mV/div (80 mV full screen), with no applied signal and without filtering, averaging, or HiRes at 350 MHz bandwidth. These values are good, and they match the information in our data sheet. The input-referred noise floor of the acquisition path is significantly lower at the 10 mV/div setting. So the change in the DVM reading is expected even though the external input signal itself has not changed.
Best regards,
Andre