Ok, you have proven that your scope can properly do a differential measurement.
Meaning if you place channel #1 on the VCC and #2 on the GND with this setup, the MATH channel will show your true noise at the output.
Also, looking at the current background noise in the MATH channel, your scope's approximate background noise level is ~15mv p-p. It might be possible to turn on the scope's measurement stats for the MATH channel as well.
Now, your scope's timebase is 10ms/div and the waveform is approximately 2 divisions long.
IE: 1s/0.020 = 50Hz. Is this correct?
If so, this interference is not your switcher oscillating, it's a leakage from your AC mains bleeding into/through your circuit. Changing a tiny output cap or inductor on your PCB will not solve this problem. (Those changes are to correct problems at the 2MHz switching speed which we cannot currently see.) Now that your scope is properly setup, please do the differential measurement of the rail, with both probe GND clips tied together, then clipped to the dreamcast's GND through a series 100ohm-up to-1kohm resistor. The purpose of this measurement is to verify that the route of this 50Hz interference isn't coming from a looping GND throughout your multiple AC main power supplies throughout your measuring scope's GND. Verify that with the same time-base setting that your competitor's board doesn't bleed through the same 50Hz.
(You might as well test the 12v rail too. This interference may be coming through there.)
If your video output is 50Hz, this may be CPU processing load once every V-sync. To verify this, load a different game and this waveform's duty cycle should change or disappear. If this is the case, then we need to look at the regulation feedback filter on your PCB and again, the same test is worth looking at on your competitor's PCB. Also, this 50hz signal may not be on the 5v rail if it is due to CPU processing modulating current draw when rendering each frame.