-1: Yes, danadak, the screen is showing A * B for the math trace, not A + B.
0: The Rigol DS1054z has a known issue with RMS calculations if one is using certain older firmware revisions. This has (probably) been fixed in the latest firmware revision. So we need to make sure that Hobgoblin is using the current firmware in the scope. This is 00.04.04.01.01.
1: The Rigol's Measurements are calculated from the Screen Display data, so partial periods shown on screen can influence the results. To get the best accuracy from the left menu Measurements, one should display many full periods on the screen (say, 10 or so rather than 1 or 2) and display them at the greatest height (lowest V/div settings) possible while still keeping them completely on-screen without vertical clipping.
2. The correct desired measurement for the power here is probably Average Power. However in the Audio field, "RMS Power" is an often-cited measurement, even though it isn't really meaningful. See
http://www.eznec.com/Amateur/RMS_Power.pdffor an excellent explanation of this issue.
3. As the above linked document shows, Vrms * Arms = Pavg.
4. So.... let's first confirm that we are running the latest firmware that calculates RMS approximately correctly. Then let's display a larger number of wave periods on the screen, with the waveforms displayed at a V/div setting that maximizes the height on screen. Since we have sine waves, the number we get from simply multiplying the Vrms x Arms values shown should converge on the value we see in the Math Average measurement window. Maybe. Hopefully.