Yes, right, the difference is way too much to be explained by different sample rates. Even though the bandwidth is much lower with the short memory (hence low sample rate) in average mode, all aliased signal/noise content still adds to the result.
Of course I have been curious and when I tried to reproduce a similar situation on a Siglent SDS1104X-E, I only got about 120µVrms noise in normal acquisition mode (without averaging) which wouldn’t be a good candidate for demonstrating further noise reduction by averaging.
At 1V/div, one LSB of the ADC equals 40mV and the measurement hints on a rather stable LSB that only toggles once every ~300 samples on average.
If I got you right, then the Keysight EDUX1002A shows ~20mVrms which means the LSB would toggle for every other sample.
Even though this is not the primary topic here, I should explain that one reason for this huge difference might be that the 1V/div is a sweet spot for the SDS1104X-E, because at that setting, the first attenuator is in effect and the VGA operates at minimum gain. In this scenario, there is very little analog noise to trigger the (much higher) granular noise of the ADC.
With average acquisition mode it gets a bit tricky.
Using average acquisition mode on the SDS1104X-E doesn’t change much. As it seems, there is just some numeric base noise that cannot be undercut, probably due to limited resolution of the measurements and/or math. Not a problem at all, just to be kept in mind.
On the Keysight EDUX1002A, the averaging kills all the granular ADC noise and the measurement falls back to a similar numeric noise as on the Siglent – which is quite baffling, to say the least.
I’ve long thought about that in order to find a plausible explanation and the best guess I can come up with is that the Averaging mode on the Keysight EDUX1002A isn’t meant for resolution enhancement, hence the averaging result is truncated to 8 bits. Only this would result in a threshold effect as you’ve observed.