@w2aew
Alan, the "Auto" function on the Rigol actually does setup the trigger parameters. It's not the same as the trigger's "auto" mode.
The signal is only jittery when it's close to (and larger than) the full screen height. When it's about half-height or smaller there's no problem.
I made a video of the problem:
http://youtu.be/piO2V77z7uA
It starts off with a triggered and locked signal. Then I change the vertical scale and it loses lock. I move the signal into view and show that the trigger is in the middle. Then I change it to a square wave and it locks. Again I show the trigger location. Finally, I change it to a triangle wave and it loses lock again.
OK, I was referring to the AUTO Trigger mode, not the Auto-Setup, which is what you were referring to. I understand that the auto-setup does it's best to setup the vertical, horizontal and trigger.
I think I now know what it happening with your setup. The Instek SFG-1003 is a DDS type function generator. DDS stands for Direct Digital Synthesis. This means that the waveforms are created using a DAC (digital to analog converter) from data samples produced by an FPGA. It is possible (and likely) that the waveform produced by the generator has small "wiggles" on it due to the finite number of voltage levels/steps that the DAC can produce. The DDS usually with dither the samples a bit to smooth these out over the long term. The waveform frequency is adjusted in most DDS generators using a phase accumulator - which has the effect of decimating or replicating samples. This generation process creates waveforms that have a bunch of small "micro-steps".
However, your new scope has enough bandwidth to actually SEE and trigger on these small little steps. So, for the waveforms with "slower" edges like the sine and square wave, these microsteps will be more apparent, and the scope is triggering on them which is what is causing your instability.
You can verify this a couple of ways. Try engaging the vertical bandwidth limit (20MHz?) on your scope if it has it. That may filter the micro-steps enough to smooth them out so that it triggers cleanly. The other way you can verify this is to zoom way in on the waveform and do a single acquisition - you will likely see these microsteps. I have a Leader function generator that is also a DDS, and the wider bandwidth scopes "see" the same thing that your new Rigol does.