It also varies with signal level and volts/div setting which indicates that something uncouth is going on.
I think the display also varies with the spectrum of the incoming signal. Didn't properly tested yet if the spectrum outside 100 MHz is the real cause, but I noticed that a 30 MHz square wave with a 10ns rise/fall time is displayed reasonably well, while the same 30 MHz and same voltage level from a TTL output (probably a 2ns fall/rise time, not specified in the generator's specs) is very distorted in various ways depending on the on/off status of the sinx/x mode and vector/dots mode. The slow edge signal was taken from the signal output of a DG4102 DDS generator, while the fast edge signal was taken from the sync output of the same generator which sync output is described as TTL compatible.
I didn't double checked with an analog oscilloscope if the two signals were really the same square wave with the only difference in edges' speed, but I'm almost sure they were the same, because the artifacts I've seen were nothing like normal ringings or reflections. They were weird and unnatural looking shapes that I've never seen before, clearly visible only when the fall/rise edges were << 10ns (for a 30 MHz square wave, 50%, TTL level, and the oscilloscope sampling rate at 250MSa/s).
Here's an example with sinx/x off, dots mode, run mode, for the same 30 MHz square signal except the second one has sharper edges:
Please note that we see so many dots because the oscilloscope is in run mode and many screens are overlapped, while the generator and the oscilloscope does not have the internal clocks synchronized. The actual number of samples for a single trace is very small:
Same 30 MHz slow vs fast edges, this time with sin(x)/x on:
Maybe I'll redo the test in a more controlled way and take some snapshots.