Just an update on the final (so far) design, just in case anyone stumbles on this later.
It is basically the same stuff Siglent and others do, but without an encoder. I have 4 buttons - left, right, up and down. Left and right select a position (digit), and up and down change values starting from that position.
With this method, the rate of change can be predictable and reasonably slow. If you want to go faster - just go to a higher order of magnitude.
Another major benefit, which I actually use quite a bit - it allows for monotonic manual sweeps with arbitrary step.