Try to understand Standard Deviation readings

[quarks]

Hello all,

a while ago I saw on my Tek DMM 4050 in ANALYZE STATS Mode SD 0.00000 for the very first time. Statistically I did not and still do not understand that. I tried several times to repeat the measurement and saw it 2-3 times happen again. Each time just before it shows 0.00000 SD was around 100n. I had a short discussion with robrenz (and if I remember right he never saw this on his 8846). So I came to the conclusion that it may be a threshold just under 100n that makes the TEK show 0.00000. After that I forgot about this behavior.

But right now, while testing some new lab gear, I saw STD Deviation 0.00ppm also on one of my other DMMs.

Anyhow it remembered me to the above TEK behavior. So I checked some older pictures and  found SD values below 100n and even with 200p on the 4050. Now I guess my conclusion was wrong.

I checked some documents Fluke and TEK about the STATS function and here is also a picture (Fluke_Figure1) from Fluke App Note "Understanding specifications for precision multimeters".

Does anyone know more about this and can explain it?

thanks
quarks

[KJDS]

Looking at the top picture, it shows a min of 94, a max of 100 and a mean of 99. It therefore isn't following a normal distribution, certainly for whatever produced the min reading.

It's possible that the max is really at the +3 sigma point of 100 and the mean is at 99, giving a stdev of 0.3 which gets rounded to 0

[alm]

Or it could just a very narrow distribution with for example 2877 data points at 10.00000 or 9.99999 and 1 outlier at 9.99994. Without knowing the distribution, there's no direct relationship between standard deviation and min/max.

It would be nice if the stats mode would offer resolution beyond that of the ADC. Some meters do, but apparently not the Fluke/Tek.

[quarks]

thanks for the replys.
According to the reading and if I remember right the 10V Reference / DMM 4050 combination did not change much (below 1µV over hours) that is why AVG is only 1µV under the nominal 10V.
STATS resolution is 7 digits.
Here is another picture which shows equally distributed figures (but this time I trimmed the 10V to just stay under 10V to see max. 7 digit resolution in the Tek 4050 STATS mode with all MIN, AVG and MAX values)

[quantumvolt]

As said in posts over - what you see is a skewed distribution. The more skewed it is the closer the mean to the maximum (generally).




https://onlinecourses.science.psu.edu/stat504/node/14


In order for the mean to be 9.999 999 in the interval [9.999 994, 10.000 000] there must be "in average" 5 times as many high readings as low (think of the 94 weighing in (94-99)= -5 and the 100 gives (100-99)=1).

Your 10 V ref is too good for your meter (what is the source?). I guess your meter's 1 microvolt reading is significant just as it is for the 34401A in stat and RS232 readings. But the span of the readings is only 6 microvolt, so 1 microvolt resolution is not enough. Also there might be rounding errors from the instrument's math. Display rounding is not important because the truncation of results happens after all math is done.

The only way to find the real standard deviation (which by def. is diff. from zero as long as at least two readings differ) is to use that nice USB plug and get the data for processing on your PC. The calculations will though be somewhat meaningless because all observations will be grouped in a "histogram" with only 7 columns from 94 to 100. This and inherent noise already in the readings means that it's a chance game to bin any single given reading. The power of the setup is that you have thousands of consecutive readings in a very narrow interval. Good source, good meter and a nice setup.

[alm]

A limited resolution can be worked around by oversampling (averaging lots of samples). The resolution should increase with the square root of the number of samples, assuming no bias in the sampling/rounding. As long as the data is distributed over at least two bins, then there is sufficient information to calculate an accurate average given enough samples. That is why I mentioned that an increased stat resolution would be useful. I've seen some meter give results in the sub-nV range when averaging the shorted reading over long time periods. But those meters were designed with a beefy 8-bit MPU, unlike the Fluke which probably features a whimpy 32-bit MPU that needs to run a full-blown OS in addition to complex stuff like double precision floating point math.

[quantumvolt]

Or it could just a very narrow distribution with for example 2877 data points at 10.00000 or 9.99999 and 1 outlier at 9.99994. Without knowing the distribution, there's no direct relationship between standard deviation and min/max.

It would be nice if the stats mode would offer resolution beyond that of the ADC. Some meters do, but apparently not the Fluke/Tek.

In this case t is impossible for the situation you describe to happen. If you want a single reading to draw down the arithmetic mean by 0.000 001 it would have to be quite low (screaming error). To make it simple - let's say 2877 times 10 and  1 time V? - your mean will be [(2877x10)+(1xV?)] / 2878 = 9.999 999. Solve for V? = (9.999 999 x 2878) - (2877 x 10) = 9.997 122 (check numbers for yourself).

Furthermore - even if engineers think they can measure themselves to almost anything by all kinds of fancy stuff, they tend to forget that it is mathematics that lies behind it all. In this case the math says :

Note that this averaging is possible only if the signal contains perfect equally distributed noise which is enough to be measured by the A/D converter. If not, all 2^n samples will have the same value, the average will be identical to this value, and the oversampling will have no effect, so the conversion result will be as inaccurate as if it had been measured by the low-resolution core A/D. This is an interesting counter-intuitive example where adding some dithering noise can improve the results instead of degrading them.  http://en.wikipedia.org/wiki/Oversampling

In my backyard there is nothing perfect or equal nor is there anything continuous. If there were, I could measure picovolt with a 0 1/2 Digit meter. The big point however is that this result applies only to the the mean. The mean is a measure of location and can be estimated as a linear interpolation between the values of the bins over and under the unknown mean. The standard error of the mean decreases with increasing sample size. But the standard deviation is a  measure for spreading in the sample as a whole and requires all bins.

" ... the variance of the mean decreases when n increases. This formula for the variance of the mean is used in the definition of the standard error of the sample mean, which is used in the central limit theorem."


https://en.wikipedia.org/wiki/Variance#Sum_of_uncorrelated_variables_.28Bienaym.C3.A9_formula.29

[alm]

In this case t is impossible for the situation you describe to happen. If you want a single reading to draw down the arithmetic mean by 0.000 001 it would have to be quite low (screaming error). To make it simple - let's say 2877 times 10 and  1 time V? - your mean will be [(2877x10)+(1xV?)] / 2878 = 9.999 999. Solve for V? = (9.999 999 x 2878) - (2877 x 10) = 9.997 122 (check numbers for yourself).

What are you talking about? These are the constraints:
Min = 9.999994
Max = 10.00000
Avg = 9.999999
SD = 0.00000
n=2878

Assume the dataset 9.999994; 2876 * 9.999999; 10.00000. Rounded to six decimal places, this set has a mean of 9.999999 and a standard deviation of 0.000000095 (=0.000000). There could be substantially more 10.00000 datapoints before it would raise the SD to > 0.5 uV, but I don't feel like calculating this and it's not very relevant to the discussion.

Furthermore - even if engineers think they can measure themselves to almost anything by all kinds of fancy stuff, they tend to forget that it is mathematics that lies behind it all. In this case the math says :
[...]

Which statement of mine are you contradicting? The one where I stated that the data has to be distributed over at least two bins?

In my backyard there is nothing perfect or equal nor is there anything continuous. If there were, I could measure picovolt with a 0 1/2 Digit meter.

Of course the only thing you're improving by oversampling is resolution, not accuracy (unless your accuracy was limited by the resolution). You could get 1 pV resolution if you where patient enough.

The big point however is that this result applies only to the the mean. The mean is a measure of location and can be estimated as a linear interpolation between the values of the bins over and under the unknown mean. The standard error of the mean decreases with increasing sample size. But the standard deviation is a  measure for spreading in the sample as a whole and requires all bins.

Since most meters are integrating ADCs, you're already averaging over the integration period. If the noise of the DMM is negligible compared to the source and the noise is evenly distributed over the frequency spectrum we're interested in, then increasing the integration period by a factor of 10 is the same as averaging over 10 samples. In the first case the standard deviation is decreased by a factor sqrt(10), and in the latter case the standard error of the mean is equal to sigma/sqrt(10). So you can only compare results for measurements with the same integration time and averaging, even if you have an ADC of infinite resolution. If you're interested in the average voltage from a voltage source, then the spread of the average is the correct measure.

If you want to quantify the noise from the voltage source, then you'll need to specify a noise bandwidth, and then you might into trouble that the reduced bandwidth of your oversampling setup is not sufficient.

[robrenz]

A limited resolution can be worked around by oversampling (averaging lots of samples). The resolution should increase with the square root of the number of samples, assuming no bias in the sampling/rounding. As long as the data is distributed over at least two bins, then there is sufficient information to calculate an accurate average given enough samples. That is why I mentioned that an increased stat resolution would be useful. I've seen some meter give results in the sub-nV range when averaging the shorted reading over long time periods. But those meters were designed with a beefy 8-bit MPU, unlike the Fluke which probably features a whimpy 32-bit MPU that needs to run a full-blown OS in addition to complex stuff like double precision floating point math.

Were you taunting me with this?
8846A manually ranged to 10V, 100NLPC digital filter on, shorted inputs stat mode.  This is actually hundreds of 0 readings at 10nV resolution with just 2 spurious spikes of +/-1.79µV  10nV on min-max of 10V range and 1fV on average and 1pV on SD is not enough resolution? 10nV on 10V is 1 part per billion resolution.


Trend plot of 3 minutes of zero readings at 10nV resolution


Stats of same thing


Trend plot showing spurious spike and proving that the resolution is indeed at least 10nV on the straight line above.
Fluke tech is getting me an answer about what the spikes could be from.

[quantumvolt]

Or it could just a very narrow distribution with for example 2877 data points at 10.00000 or 9.99999 and 1 outlier at 9.99994. Without knowing the distribution, there's no direct relationship between standard deviation and min/max.

...


What are you talking about? These are the constraints:
Min = 9.999994
Max = 10.00000
Avg = 9.999999
SD = 0.00000
n=2878

Assume the dataset 9.999994; 2876 * 9.999999; 10.00000. Rounded to six decimal places, this set has a mean of 9.999999 and a standard deviation of 0.000000095 (=0.000000). There could be substantially more 10.00000 datapoints before it would raise the SD to > 0.5 uV, but I don't feel like calculating this and it's not very relevant to the discussion.

My mistake. I never saw that you wrote "10.000 000 OR 9.999 999". So when I wrote "a single reading" I was thinking 2877 and 1. But your original statement opens up for 2876, 2875, ...., so the situation is indeed possible. Sorry.


When it comes to engineer chat, I am out. The situation is imo purely physical / mathematical.

If anyone is interested, let's say we have a perfect constant nominal 10 V source with unknown real value Vref. We also have a perfect meter that only gives two readings without ever erring: The input is less than or greater than 10 V. As the meter cannot display "equal to" 10 V - if Vref is exact 10.0 V the meter will never output anything at all. If Vref is LT/GT then 10.0 V the meter will output the same result forever. Still we now nothing more than which side of 10.0 V the Vref value is situated.

Now add a perfect noise source (equal distribution in values). Waiting long enough, the value of Vref can be measured (estimated) to any accuracy from the relationship of LT/GT readings. This in spite of the meter's ultimate low resolution. The process is guaranteed to converge. In this - and all other representations of the two values/bins case, it is meaningless to talk about spread (as in variance and standard deviation).

In practical measurement the above principle can be used to decrease the standard error of the mean. But the uncertainty of any single reading is governed by the stochastic parameters of the underlying model and cannot be improved by any means (without changing the setup). The spread measures are attempts to aggregate the distances between measured values. In order to do this without introducing any more uncertainty one must categorize the data in a way that determines one and only one bin for every possible reading of the meter (resolution). In this situation it is impossible to get a standard deviation equal to zero unless all readings are equal.

That is all I can say. I guess I withdraw from this discussion 

[quantumvolt]

PS. In the perfect meter case we are not allowed to infer 'no output means exactly 10 V".

[alm]

Now add a perfect noise source (equal distribution in values). Waiting long enough, the value of Vref can be measured (estimated) to any accuracy from the relationship of LT/GT readings. This in spite of the meter's ultimate low resolution. The process is guaranteed to converge. In this - and all other representations of the two values/bins case, it is meaningless to talk about spread (as in variance and standard deviation).

Adding noise is a perfectly valid technique. I know at least one Datron DMM that uses this exact technique to guarantee increased resolution by oversampling. I think they even vary the amount of noise in steps smaller than one bin to increase the resolution even further.

In practical measurement the above principle can be used to decrease the standard error of the mean. But the uncertainty of any single reading is governed by the stochastic parameters of the underlying model and cannot be improved by any means (without changing the setup). The spread measures are attempts to aggregate the distances between measured values. In order to do this without introducing any more uncertainty one must categorize the data in a way that determines one and only one bin for every possible reading of the meter (resolution). In this situation it is impossible to get a standard deviation equal to zero unless all readings are equal.

My issue with this is that your underlying model (the DMM) is itself sampling and averaging a stochastic process (the voltage source). If the meter is integrating the voltage over 1/60th second, is this a single sample? What if you set the meter to integrate over 1/6th of a second? The variance of the latter will be lower (with our perfect DMM and perfect noise source). Which of them is the inherent variance of your process?

[robrenz]

Marcus, please try this.  Manual 10V range and apply your 10V signal and zero the meter.  Then do a trend plot.  Trend plot will keep the math zero.  (going to stat mode cancels the math zero)  Then you should have 10nV resolution on trend plot of the variation from the voltage when you zeroed it. I realize this does not directly answer the SD question but may be interesting. I would do this myself but I don't have a 10V reference stable enough.

[quarks]

I started it (see first 21 min. and 43 min.). How long do you think it should run?

edit: had to stop it after 77min.
edit: because on the next test it looked different, I searched for the cause and found that the D-Filter was on!
        With D Filter off,  all other measurements showed flat lines with few spikes.

[robrenz]

If you could do the shorted input under the same conditions and get a picture of the very first screen (sweep) of trend plot before it starts its second sweep to see the frequency of spurious spikes.  Then do the same with the 10 volt input and get a picture of the very first screen (sweep) of trend plot before it starts its second sweep.  Now when you compare those two pictures you should be able to get a feel for what is general short term variation of your reference and what is the spurious spikes of the 4050.  If you can collect the output of your "better" meter on a pc, you could set it to 7.5 digit, fast integration to get 64PLC which would be as close as you can get to the 100PLC of the 4050. Then capture and plot 1.56 times as many samples with 10V input as the samples of the single trend plot trace and then compare all three charts.

[quarks]

If you could do the shorted input under the same conditions and get a picture of the very first screen (sweep) of trend plot before it starts its second sweep to see the frequency of spurious spikes.  Then do the same with the 10 volt input and get a picture of the very first screen (sweep) of trend plot before it starts its second sweep.  Now when you compare those two pictures you should be able to get a feel for what is general short term variation of your reference and what is the spurious spikes of the 4050.

I did what you suggested with DMM4050.
Here is my Setup
Reference and Tek warmed up for >4h
Tek Man 10V range, Filters off, 100PLC, HighZ
1. shorted
2. 10V applied, zeroed, than trendplot

The shorted I did several times, to see if it is always totally flat (because of manual 10V range).   
Then there were very few spikes (like in the 10V zeroed) in the first screen and more in the later screen.

It looks like every spike is comming from the 4050 itself.

bye
quarks

[robrenz]

Thanks Marcus,  Those shorted plots look very similar to mine. I am going to try to setup my 8846A connection to my laptop and gather some similar data as the trend plot and see what that looks like.

[quarks]

Hello robrenz,

what do you expect from the PC connected? Are there more advantages than the obvious edit convenience and better looking graph?

BTW  to zero the source value and then trendplot  was a very good idea. This seems to be the perfect way to check the stability/drift of sources with great detail.

Bye
quarks

[robrenz]

what do you expect from the PC connected? Are there more advantages than the obvious edit convenience and better looking graph?

I thought I might be able to access the very high internal resolution that we see in trend plot.  But you only get one digit extra resolution on data saved to memory or transferred to pc

Edit:  I can see why they don't do that.  10nV resolution on 10V is a 9-1/2 digit meter.  As far as I can tell the trend plot resolution on the 10V scale is 10nV at any integration time (NPLC).  That implies the high resolution is not coming from extra long integration time or averaging but that it has at least a 30 bit adc.  The noise level and reading stability definitely improves with increased integration time which makes sense.  Does this make any sense or am I missing something.