My gain unpack code is currently using "digit>8" not "digit>=8". I assume that I should switch this to "digit>=8"?
yes, otherwise the raw constant "88888" wouldn't decode/unpack properly. As you saw, it corresponds to a gain of 0.911112 which is validated by many tests now.
Maybe it would help to think about how the gain is applied by the stock firmware: it doesn't convert the constant to a fixed-point decimal number like what we're doing. That would be incredibly wasteful on that mcs-48 architecture with no hardware multiply. Here's roughly what it does, starting with a raw reading 'R' , with for example a gain constant K=90700. I'll call each digit Kn such that K1 = 9, K3 = 7, ...
C = R; //C will be the "almost calibrated" reading, just before the offset is applied.
K1 = 9;
// At the "1" digit, we're working at the 1/100 decimal position, so R is "decimal-shifted" 2 digits right.
// Since K1 >= 8, we need to subtract (16-9)=7 times the shifted raw reading.
C = C - ((16-9) * R/100);
K2 = 0;
//do nothing
K3 = 7:
// digit 3 : means "add 7 times", but now R is shifted 4 positions
C = C + (7 * R/10000);
//and the rest of the digits are 0, so do nothing.
//The equivalent operation is C = (1 * R) - (7 * 0.01 * R) + (7 * 0.00001 * R)
//can also be written as C = R * (1 - 0.07 + 0.00007) or
//of course, C = R * 0.9307
The reverse process is sortof like a long division I guess ? I think it goes something like this. Assume all offsets are 0 for simplicity. The meter has a raw ADC reading 'R', and it knows the expected calibrated value 'C'.
- calculate error 'e = c - r' . Can be negative of course
- shift raw value right, i.e. divide by 10. First iteration needs to start at a 1/100 division. let's call it 'sr = r * (1/100)'
- how many times does 'sr' fit in 'e' ?
- if it fits 0 to 7 times, then that encodes directly to digit 1.
- if it fits -1 to -8 times, then "encode" that digit as (16 - <value>), so 0x0F for -1 to 0x08 for -8.
- if it fits 8 or 9 times, problem : those digits indicated negative values, so we need to cheat:
carry 10 to the digit to the left (by incrementing by 1 - careful if the digit was already 7, you'll need to do the same manoeuvre recursively), and set the current digit to (value - 10) i.e. -2 or -1.
In other words, use the fact that 8 * sr = (10 * sr) + (-2 * sr) - if it fits -9 times, same idea : borrow 10 from digit on the left, adjust, and continue.
- calculate new 'e' value for next loop, i.e. remainder. Right shift sr again and continue
I think that should work, conceptually at least. Interesting consequence of the +7/-8 limit on digits : some gain values can be encoded in more than one way, for example 0001C and 00006 both correspond to a gain of 1.000006 .
Also, I'd still like to know what range check to use for the gain. What limits should I set for user entered gain values?
That would be the two extremes we've tested,
77777 1.077777
88888 0.911112