Resolution (digits) vs accuracy (HP 4274A/4275A)

[Nevada]

Hi,

Is there any sense to use the HP 4274A/4275A LRC meter in "high-resolution" (5.5 digits) mode despite it is specc'd "only" to 0.2%? E.g. if the indicated value is 1.00000uF, then the error is +/-0.002uF, so the nominal value could be somewhere between 0.99800 - 1.00200uF. It seems to be it is more closer to a 3.5 digits precision than a 5.5 digits.

[Kleinstein]

The higher resolution could still be used for things like matching capacitor sets or seeing of capacitance changes with temperature or similar. With higher resolution it is quite common that meters have better resolution than accuracy. The ration from accuracy to resolution also gets better lower in the range, e.g. just before one can switch to the next smaller range.

The resolution specs could be for the actual display resolution or a noise level to get 1 count of RMS noise for the relevant speed. The accuracy specs are in a way to essentially guaranty that the error is smaller. This usually mean an additional factor of 2 or 2.5 for actual scattering of the true value. So even of the accuray would be limited by the noise, the accuray specs would be some 2.5 times higher then the resolution number. So the resolution is kind of requited to be at least 2-2.5 times better.

[alm]

That's very common: for example the venerable HPAK 34401A is 6.5 digits, so on its 1 A AC range it will have a resolution of 1 µA, but the uncertainty even within 24 hours after calibration is 0.1% of reading + 0.04% of range, so for a 1 A input the uncertainty would be 1.4 mA, or 1400 counts.

You should distinguish precision, resolution and accuracy. See here for a nice, graphical illustration of these concepts. The reason why resolution has value beyond accuracy is that often the linearity and noise floor is much better than the accuracy. So you may not be able to say if a particular cap is 0.998 µF or 1.002 µF, but you might be able to know that a cap that measures 0.9984 µF is larger than one that measures 0.9983 µF, despite not knowing the value of either cap accurately.

[mendip_discovery]

Also be aware stuff swaps resolution for speed. It's a trade off, not everyone wants to wait 5 seconds to get a reading.

I have found some stuff goes for amazing resolution but the last few digits are just wasted as the unit wanders all over the place. Looks are the Keysight handheld dmm.