Their application note is using a detector/ AM demodulator on the amp input to get an audio-frequency input, and a tap on the RF output. You say you are using two taps though. With two signals of the same frequency (the RF in this case) in XY mode youre going to get anything from a diagonal line, to a circle, depending on the phase.
I think he means the RF-D "tap" which also demodulates and RF-S which is an RF sampler.
I would first verify that the scope is set up properly in XY mode. The instructions on the website are written for an analog scope and the "helpful" advice to use Tek 2213 points towards that too. However, there is no reason this sort of measurement couldn't be done on the Rigol too.
I would set the scope in XY mode first.
* Menu button above the time base knob, select timebase "XY".
* Make sure channel 1 and 2 are selected as sources (same menu)
* Enable channels 1 and 2 using the channel buttons, set coupling to DC on both (right hand soft buttons)
* Set sensitivity to 2V/div using the voltage level knob (next to the channel buttons) on each channel (press channel button, turn knob, press second channel, turn knob)
* In the same channel menu set the probe to "1x" if you had "10x" selected (the taps are connected by coax, there is no 1:10 divider in them). If you don't do this, you will see 10x higher voltages shown on the scope but it shouldn't really matter for this measurement - just make sure both channels are the same.
If there is nothing connected to the inputs you should see a dot in the middle of the screen. Now touching the probes should draw a line, either horizontal (if you touch CH1 probe) or vertical (if you touch CH2).
Connect your taps and you should be able to see something. You probably won't see a nicely filled trapezoid like that because the Rigol doesn't have so much phosphor persistence as an analog scope (and you can't turn persistence on in the XY mode

) but you should be able to see something at least. You may have to turn on the brightness too. What is important is the outside shape, not so much the fill.
I haven't tried this measurement myself but I have the same scope.