Hi!
I'd begin with looking at as many of the power supply pins to the various analogue chips on the main/signal PCB as you possibly can!
To do this, write down all the device numbers on a piece of paper, then download the data–sheet for each, then print out the page showing the Vcc and GND pins of each, usually the first page.
Remember that in equipment of this quality and complexity, the power supply rails that you can identify on the PSU may not feed all chips direct, there could be additional regulators on the main/signal PCBs, which could be either linear or switch–mode type!
What you are looking for is:–
a) Adequate supplies to each device, both for magnitude and freedom from extraneous noise or ripple;
b) Split (positive and negative) DC power supplies correctly balanced with respect to zero.
There's something distinctly wrong with these waveforms you've shown us that points to an output driver/amplifier issue, as the amplitudes are negligible and the distortion/noise you're seeing is due to the oscilloscope acquisition–stage I/P amplifier stages going flat out to read what little there is!
The 50MHz waveforms I estimate to be 5–7 mV RMS which'll simply get lost in general hash, noise pickup on connecting–leads, etc., etc!
I would have thought such a generator to be capable of 100mV RMS at least over it's full rated frequency range!
Give my suggestions a try and report back to us what you find!
Chris Williams
PS!
I don't know how complete it is but I believe Artek can supply the S.M. for the Anritsu MG3631/3632 – there's certainly a pic of one of their CDs they make for that model – I think Artek supply it both as download or cd, and the cost, about £17–£20 (same in USD!) is well worth it – Artek do not supply Chinese–scanned or photocopied rubbish!
PPS!
I've just checked Artek's site –an S.M. with schematics is available for the MG3631a/MG3632a for $12.50 – buy yourself that and let us know what you find, please note – you'll need to find a means of extracting sections of schematics (print/scan or use a free pdf extract tool) to post in order for us to advise please!