unlikely a hardware problem.
Is it as bad when selecting GND instead of AC/DC? If no then you’re looking at thermal noise in the 1Mohm input resistor.
Try to limit the input bandwidth from 500 MHz to a lower settings: does that improve things?
You did run SPC and that passed ok?
Look at the spectrum (option Math, FFT): is it white noise, pink, or some distinct peaks? You may be sitting in a noisy environment, nearby radio transmitters, cell phone towers, WiFi APs, overhead power lines, VFDs and so on…
Hello sicco!
AC/DC/GND, or 1M/50 ohm, doesn't matter really, it is more or less the same in all modes.
With Envelope acquisition mode, and measuring Pk-Pk, on channel 1, 1.00 mV/div, DC-coupling, it after some 15 seconds gets:
Full Bandwidth: 1.60 mV
Bandwidth 150 MHz: 1.40 mV
Bandwidth 20 MHz: 860 µV
Yes, the SPC has passed 6 times or so. At one occasion it didn't and said it needed a repair, but when just running it again everything was OK.
I would call the FFT pink (see attached picture), with the strongest peaks on channel 2 for some reason, at worst about -100 dB, peaks at 220 MHz, 300 MHz, 525 MHz, 600 MHz, 680 MHz, 750 MHz, 825 MHz, 900 MHz, 1.14 GHz.
Actually, the peaks get attenuated when the channel is set to GND instead of DC or AC, more attenuation at higher frequencies (see second picture).
It could be a noisy environment, but I haven't seen this problems on other scopes on the same place (TDS6xx, TDS7xx, Rigol DS1104Z, ...).
Thanks for your help and advice, much appreciated!