I finally got things in my little research project to the point where I would consider it "not terrible."
Just to remind everyone, for useability testing I have a mock interface to change a voltage between the values of 1.250V and 30.00V, using just a 20PPR rotary encoder. The challenge here is to give the user the ability to change the setting as conveniently as possible--avoiding overshoots and not requiring the user to spend a lot of time spinning the encoder just to make large changes.
One of the important changes I made was to basically throw away half of the actions (=clockwise or counter-clockwise click) and turn the encoder into a 10PPR encoder instead of a 20PPR encoder. The problem was that it is actually too hard to control at 20PPR. It's too hard to keep up with the changes, that is.
Note that, from the standpoint of calculating velocity, I still count the actions that are otherwise thrown away. This gives better resolution.
I still have the three step sizes that I had earlier: 1mV, 10mV and 100mV. The problem with my earlier implementation was that I required the user to maintain the velocity to stay at particular step size. This meant that the user (=me) couldn't keep up with the changes for the larger step sizes and would, more often then not, overshoot.
I changed it so that the user can slow down once they have stepped-up the speed. The step size can now only increase once the user starts spinning the encoder. That is, they can step up but not down. The encoder has to come to a complete stop before the step size returns to the 1mV size. This actually works out pretty good for this particular mock environment. (Setting an output voltage.)
Another change that helps a lot is that when the step size is 10mV then the last digit of the setting is always "0", and if the step size is 100mV then the last two digits are always "00". This helps the user focus on what is actually changing. (Like Christe4nM suggested.)
There are two velocity thresholds to enter into the 10mV and 100mV step sizes. The user has to be fairly determined to pass these thresholds. Once they are passed, however, the user can slow down and catch back up and regain control. Control is one of the keys.
Predictability is another key. So far as the user having a pretty good expectation as to when they are going to "step up." That doesn't seem too bad currently but could use some improvement.