Then I became aware that "fuzz" was actually a small waveform. Then tried grounding the probe... and still there.
What did you think it was?
Anyway, my DS2202 produces a similar-looking waveform with a grounding cap and all the dials turned to the extremes like you have (that's 500uV/div actually, not 5mv/div.) I don't want to be "that guy" desperately defending his purchase, but the set of problems that can't be solved by a) setting bandwidth limit, b) using higher level signals, or c) using more gentle timescales appears to be a small one; certainly one I haven't encountered after using the scope for many years now.
One thing that
is disappointing, though, is that Rigol don't seem to specify a noise value or figure in their datasheet. Not sure if it's standard practice to omit such a figure from the specs, but it certainly makes it harder to defend the performance of a product when they don't make a specific claim.
Noise is an inevitability of using wide bandwidths, although my attempts to calculate a figure have given an unexpectedly high result. A plain old resistor, such as the termination resistor in the scope, of 1 megaohm at 20 degrees C produces 1.8mV RMS white noise when observing over a 200 MHz bandwidth. This is over 3 divisions on your scale,
RMS, which would mean that your scope is performing better than a theoretically perfect scope, which is clearly wrong. I'm not sure how the 16pF capacitance of the frontend factors in, or whether I'm missing something else?
TLDR; I suspect you should watch the EEVBlog episode debunking the claim that digital scopes are noisier than analog. It does a good job of conveying that noise is an intrinsic part of life, not some design mistake that can be arbitrarily quashed. Clearly there is some sort of design imperfection in the Rigol scope, but still cruicially important to understand what a theoretically perfect scope looks like -- because it's
not dead flat, ever.
But i can't imagine a company would design a scope such that the highest couple of gain settings show up significant noise in their amplifier.
The high gain settings are 100% useful on more sensible timescales, when using acquisition averaging, or when using the bandwidth limit. So your expectation is unreasonable here.