Indeed, I never use 500uV/div other than in averaging mode.
Rigol DS1054Z is very noisy for mV range signals, especially when compared with old analog scopes, though those were way more expensive to buy new, and not many were having a 500uV/div option.
About internal noise, my DS1054Z measures about 1mVpp noise with no probes, 500uV/div, 5ms/div, no bandwidth limit, no average and all 4 channels on. For some reason CH3 is much noisier, showing 1.35mVpp noise.
5mV/s includes the mains 50Hz hum noises, maybe some 1/f noises too, because at 5ns/div the measured Vpp noise becomes about 2-3 times smaller.
At very fast sweeps, there is also a small artifact, like a 1-2mVpp wiggle for about 10ns around the sync point. This is easier to notice in averaging mode. It is more visible on the channel used for sync. This wiggle seems to be there no matter what. It happens at 1mV/div, too, just that it shows half amplitude in the wiggling.
In term of offset deviations, on mine the traces have less than +/- 0.5div offset on 500uV/div, and half of that at 1mV/div. The offset can shift with time, in 30-60 minutes for example going from +0.5 to -0.5 div because of temperature changes.