I've changed it to 1x, terminated it, grounded it and measured noise which is a more reasonable 720uV which is spot on and in line with my old analogue unit:
If I am reading the DS1000Z user manual correctly, Vamp is not a useful measurement for noise. Vpp and Vrms should be used. It may be that the Rigol cannot accurately measure Vrms noise because of how the display is processed. Noise should not change significantly with time/div however fast time/div settings will ignore low frequency noise.
For comparison, a very noisy 100 MHz oscilloscope should have less than 200uV RMS noise (1) and a reasonable one will be more like 20uV RMS. Most but not all analog oscilloscopes have a CRT beam width greater than their input noise making input noise measurements difficult.
(1) The extra input circuits need to provide high common mode range differential inputs increase the noise of a simple single ended input by an order of magnitude.