Keep macines on minimum 30 minutes before test and before test run Rigol selfcal and maybe also R&S selfcal.
Sign gen (R&S) out (N or BNC)-------50ohm cable -------- (50ohm GOOD terminator!)-BNC Rigol CHx
R&S out ~106mVrms example 10, 50 or 100kHz and this R&S Vrms level mean that it is TO 50 ohm load IF this instrument is with 50 ohm output (there are also other impedances dependent model options. Temination and cable need match for this)
If you forget termination you see wrong level and may see also bad flatness (cable lenght, input of scope etc play together and SWR make all measurements fail (some points may be just ok but you can not know what) and you see near like double values in voltage (in low frequencies). (as can see your data)
(note: I do not know this individual sig gen details so I do not know how is this low freq end and is it trusted level in low freq. So if you want 10MHz as reference point for 0dB vertical... use it and do this initial procedure for leveling with example 10MHz (this case use 2ns/diw also for this)
--------
EDIT: (SMB100A: Level uncertainty f = 200 kHz to 3 GHz <0.5 dB) so, I recommend to take 200kHz for vertical 0dB ref point and for this initial settings for this test and in this case 1us/div is better for horizontal)
--------
Rigol
CH1 (or) CH2 only in use.
DC
50mV/div
5us/div (use 2 or 5 ns/div if 10MHz as 0dB vertical reference. This case not directly comparable to normal scope specs)
equtime
Average 8
center trig, center vertical
Check rigol with average in use that it show exactly 6div vertical. (exaxtly 300mVp-p +-2mVp-p)
If not, adjust R&S level littlebit so that display in Rigol is 300p-p (106mVrms) It is good practice not use maximum vertical area in scope. This scope have 8div vertical so it is reasonamble to use around 6 div. (of course can use what ever levels but this is example and my practice (maybe coming fron Agilent... oh no...I mean Hewlett-Packard of course and Tektronix...
)
After this, do not touch R&S output level.
Now tst conf is ok and reference is ok for measure BW and also we have 0dB reference point (10, 50 or 100kHz)
Change R&S to 10MHz (and later step up forward)
Change Rigol to
CH1 (or) CH2 only in use.
50mV/div
2ns/div (of course 5 if 1052 original in test)
egualtime
Average 4 - 8
make tests with all desired freq points between 10 to >200MHz (no need touch vertical speed). rememember wait enough averaging if average is use! (for randomnoise I recommend averaging becouse this is not analog oscilloscope)
10MHz level is typically now around 1- 3% low compare to refpoint.
Of course you can also use different test signal levels. (Vp-p = Vrms * 1,4142 * 2)
This give good picture about real BW.
(BW maybe littlebit different in 2 main bands in V/div settings. listen relay click...you know these bands. 2-200mV and 500mV - 10V also maybe some small differents in other settings)
Also if like testing, same with realtime (but sinx OFF becouse sinx in this rigol IS bad)