I got the adc->pwm generator->coil meter calibration part working.
It is interesting to the get the adc module working. I got over the offset errors fairly quickly. But soon realized that I have to correct for gain errors too: the measured ESR is non-linear vs the true ESR, and follows a quadratic equation.
So I had to correct it. Doing so without using floating point math,

, and with little space to spare.
Here is what I come up with.
//mESR error-correction
//flowing point version: mESR_ec=0.706991*mESR + 6.64e(-5)*mESR^2
//integer version: mESR_ec=181*(mESR + mESR/32)/256 + (mESR + mESR/32)*(mESR + mESR/32) * 2 * 2 / 256 / 256
uint16_t mESR_ec(uint16_t mESR) {
uint32_t mESR_32 = mESR + (mESR >> 5);
uint32_t tmp;
tmp = 181 * mESR_32;
tmp+= (mESR_32 * mESR_32) >> 6; //+(mESR+mESR/32)^2*2*2/256
return tmp >> 8; // / 256
}