Should the number of measurements be included in the measurement uncertainty?

[Dexter_Bunco]

I am running a test on a power meter where a Fluke 5720A is used as a source and a 3458A is used to measure the source. Reason for this is because the power meter loads down the 5720A so its output has to be monitored.

So measurement uncertainty on that test should be as simple as applying the 3458A's uncertainty at that value at that range. But I have seen a past technician (presumably more knowledgeable?) add in another uncertainty factor. Per my procedure, three readings are taken from the power meter and averaged. What this other technician did was take the standard deviation (of the sample) and RSS that with the standard's uncertainty.

And I admit that my knowledge isn't good enough to know if this is correct. When I tried researching this methodology, the formula looked more complex than that too. More along the lines of ( StdDev / (NumMeas ^ 0.5)) * Degrees of Freedom

So I need some wisdom from others more experienced than I am.

1) Should it be including the number of measurements I take into the test point uncertianty?
2) Is RSS'ing it with the standard's uncertianty the correct way to do that?
3) If yes to question 1 and 2, is the expanded formula I found the more correct way?

[Kleinstein]

Wether the number of measurements of the DUT has to be includes depends on wether one want the uncertainty for a single reading of the DUT or the uncertainty of the average.  So it depends.

Using only 3 readings is a rather small number to calculate the Std. dev. Usually one should use more values - more like 10-100, at least in modern times with automated readings.

[Dexter_Bunco]

I find myself torn by that answer. On the one hand, I was trained in commercial calibration with the hard mindset of time = money so the idea of each test point taking several seconds longer (totaling many minutes longer in the total time it takes to cal) is a hard no and I just shouldn't worry about it then.

But another part of me wants to actually be a better professional and say "screw it" to production over quality and expand out the number of measurements taken.

And I am at last in a point in my career were no one can actually stop me, whichever route I take.

[CalMachine]

I find myself torn by that answer. On the one hand, I was trained in commercial calibration with the hard mindset of time = money so the idea of each test point taking several seconds longer (totaling many minutes longer in the total time it takes to cal) is a hard no and I just shouldn't worry about it then.

But another part of me wants to actually be a better professional and say "screw it" to production over quality and expand out the number of measurements taken.

And I am at last in a point in my career were no one can actually stop me, whichever route I take.

Dear please god don't go the "time = money" route.  Do it right.

[CatalinaWOW]

This isn't a time vs money question.  When you think of it in those terms you are forcing your requirements on your customer.  Best answer is to provide both answers with detail explanation of what they each mean.  Acceptable answer is to provide one or the other, with a detail explanation of what you have provided.

Think through your model of what is going on.  The instrument you are testing has noise.  The instrument you are testing with has noise.  All if the following assumes those noises are constant amplitude and spectrum.  If that isn't true you are in a whole other world of evaluation.

Start with the ideal case.  Your measurement instrument has zero noise.  If you make a single measurement you have an estimate of the mean output of the meter, but have no idea other than the manufacturers spec of how much noise there is, or how that single measurement relates to the actual mean output of the meter.  As you make more and more measurements you can compute the st. dev. of the measurements and the average.  Under a broad set of usually applicable assumptions you can also estimate the error of the average (the standard deviation of the estimate) as the st. dev. of the samples divided by the square root of the number of samples. 

When you let the noise of the measurement instrument be be non-zero the procedure for average output and the standard error of the estimate is unchanged.  The problem comes in separating the instrument noise from the DUT noise.  The measurements don't provide that information.  Basically, someone, somewhere has the measure the noise of the instrument while measuring a very low noise source.  If you dump the problem on your instrument vendor you just assume that his noise spec (or the calibration data the manufacturer provided) is correct.  Otherwise you do it yourself using some low noise source.   Once you have the data, if you can assume that the DUT noise and the instrument noise are uncorrelated you can assume they added orthogonally in the measurements and derive the DUT noise.   Unfortunately the uncorrelated assumption is much harder to justify here because line noise and radiated noise from lamps and the like are not uncommon and will be correlated.

[mzzj]

Wether the number of measurements of the DUT has to be includes depends on wether one want the uncertainty for a single reading of the DUT or the uncertainty of the average.  So it depends.

Using only 3 readings is a rather small number to calculate the Std. dev. Usually one should use more values - more like 10-100, at least in modern times with automated readings.

Wether the number of measurements of the DUT has to be includes depends on wether one want the uncertainty for a single reading of the DUT or the uncertainty of the average.  So it depends.

Using only 3 readings is a rather small number to calculate the Std. dev. Usually one should use more values - more like 10-100, at least in modern times with automated readings.

'

Generally 10 samples is preferred minimum, less than that and the standard deviation doesn't give that good representation of the mean value uncertainty. This can be "fixed" with addtitional multiplier to standard distribution
https://sisu.ut.ee/measurement/35-other-distribution-functions-student-distribution
https://en.wikipedia.org/wiki/Student%27s_t-distribution

[mzzj]

I am running a test on a power meter where a Fluke 5720A is used as a source and a 3458A is used to measure the source. Reason for this is because the power meter loads down the 5720A so its output has to be monitored.

So measurement uncertainty on that test should be as simple as applying the 3458A's uncertainty at that value at that range. But I have seen a past technician (presumably more knowledgeable?) add in another uncertainty factor. Per my procedure, three readings are taken from the power meter and averaged. What this other technician did was take the standard deviation (of the sample) and RSS that with the standard's uncertainty.

And I admit that my knowledge isn't good enough to know if this is correct. When I tried researching this methodology, the formula looked more complex than that too. More along the lines of ( StdDev / (NumMeas ^ 0.5)) * Degrees of Freedom

So I need some wisdom from others more experienced than I am.

1) Should it be including the number of measurements I take into the test point uncertianty?
2) Is RSS'ing it with the standard's uncertianty the correct way to do that?
3) If yes to question 1 and 2, is the expanded formula I found the more correct way?

1. generally yes (among gazillion of other things, ie anything that has any relevant effect to measurement. IE If your voltage reference is sensitive to gravity you need to compensate for moon's pull and groundwater level..)
2. RSS is the way to go, but note also different coverage factors. Calibration results are reported at K=2 so you need to divide the uncertainty by two before RSS it with StdDev
3. expanded formula looks bit more correct, see student's t-distribution. StdDev/(sqrt(n) is "close enough" with n>10 samples.

4. suggest reading few "introduction to uncertainty calculations" or something along those lines.
ie https://www.dit.ie/media/physics/documents/GPG11.pdf

[rhb]

In the presence of Gaussian distributed noise, the effect of the noise goes down as 1/sqrt(N).

So the uncertainty contribution from random noise when  averaging 100 samples is 10% of the the uncertainty of a single sample.  Average 10,000 samples and it is 1%.  For 10 samples it is ~30% of the single sample uncertainty.

So how much uncertainty can you live with?  You can't go below the basic uncertainty of the instrument, but you can reduce the random noise contribution by averaging samples.

Have Fun!
Reg

[rhb]

Thanks for the link.  That is a very informative paper.  I particularly liked the polarity reversals as a means of correcting for thermal EMF.

If we take the uncertainty as:

Calibration uncertainty + DMM measurement circuit  aging + DMM Vref aging + Temperature & humidity changes + Gaussian random noise

The first 4 terms are the instrument uncertainty  and the noise induced uncertainty of the  last term is (Vrms_noise*Vmeasured/sqrt(N)).  Vrms_noise is the standard deviation.  So the uncertainty due to measurement noise is (Vsigma*Vmean)/sqrt(N) where Vsigma is the standard deviation.

Have Fun!
Reg

[RandallMcRee]

In the presence of Gaussian distributed noise, the effect of the noise goes down as 1/sqrt(N).

So the uncertainty contribution from random noise when  averaging 100 samples is 10% of the the uncertainty of a single sample.  Average 10,000 samples and it is 1%.  For 10 samples it is ~30% of the single sample uncertainty.

So how much uncertainty can you live with?  You can't go below the basic uncertainty of the instrument, but you can reduce the random noise contribution by averaging samples.

Have Fun!
Reg

Unfortunately, 1/f noise, which is dominant at low frequencies is not Gaussian. Hence in "Low Noise Electronic Design" by Motchenbacher & Connolly we find:

"A fact to remember concerning a 1/f noise-limited dc amplifier is that
measurement accuracy cannot be improved by increasing the length of the
measuring time. In contrast, when measuring white noise, the accuracy
increases as the square root of the measuring time."

[rhb]

Which is what  said.  I never suggested it would solve the problem of 1/f noise.  The sqr(N) *only* applies to Gaussian noise distributions.  There are a great many other probability distributions.

The situation is generally so bad I refer to the assumption of a Gaussian distribution as "sprinkling Gauss water on the problem" because I see it done so often when it's not valid.  A pet peeve of mine is seeing it invoked for quantities which are bounded at one or both ends.

I did a couple of 9 month long projects involving hundreds of gigabytes of  wireline (measurements made by lowering instruments into the well and making measurements as it is hauled up.) data from the GoM.  Porosity and fluid saturations are bounded between 0 & 100% which is *NOT* Gaussian.  Other quantities are bounded at zero. but as a practical matter can be safely treated as Gaussian because the standard deviation is so small the error near zero is negligible.  Those projects made me very sensitive to the question of, what is the probability distribution?  So long before I start doing calculations I plot the probability density functions and satisfy myself that I can justify my mathematical operations.

And EMI is *NOT* Gaussian.  So there is no substitute for proper shielding and bypassing practices.

[RandallMcRee]

Not a dig at you, rhb, but the OP may not realize what is the dominant noise source for his measurements.

All I know is that a year or two ago, I would have happily thought *all* my noise was approximately Gaussian.

We are both agreeing, right, that the answer to the OP's original question is "yes--and more details..."?

Randall

[SilverSolder]

So, is there a good way to determine what kind of noise you are looking at -  what kind of distribution it is, if not Gaussian?

Or is it possible to prove/demonstrate a "negative", e.g. that it is definitely not Gaussian, for example?

[RandallMcRee]

So, is there a good way to determine what kind of noise you are looking at -  what kind of distribution it is, if not Gaussian?

Or is it possible to prove/demonstrate a "negative", e.g. that it is definitely not Gaussian, for example?

Good question. I'm sure there are more experienced folks than me...take my ramblings with a grain of salt....but....

1/f noise is ubiquitous in nature and in electronics. It's in all opamps, for example. That's why we have the 0.1-10Hz noise broken out separately from "noise" in all the datasheets. So if you don't know, you should probably assume its there, the question is not *if* but *how much*. Chopper opamps do control for 1/f noise, however.

To make sure you need to do a spectral analysis. You can often mitigate it by avoiding a low-bandwidth measurement, see:

https://www.analog.com/en/analog-dialogue/articles/understanding-and-eliminating-1-f-noise.html

[CatalinaWOW]

Given enough measurements you can plot the probability density function and or the probability distribution function.  But there is no absolutely rock solid answer for how close they have to be to gaussian before they are "close enough".  It depends on what you are doing.

A fairly easily accessible example to demonstrate some of the problems is male human height distribution.  Clearly it is bounded at zero (actually some number much larger than zero).  And clearly can't go to infinity.  So technically it can't be a gaussian distribution.  But if you plot the pdf you find that it is very close.  Calculating the mean and standard deviation you can predicting with fraction percent accuracy the percentage of people at heights within 8-10 cm of the nominal height.  But if you use that to predict people far off of the mean, say 200-210 cm tall you will under predict the numbers not by percentage points but by factors. 

The tails of distribution are frequently where differences from gaussian show up.  And unfortunately to only way to evaluate this is with huge data sets.  But fortunately it often doesn't matter.  The error from using a gaussian assumption on human height wouldn't matter in comparing the calibrations of two tape measures, pretty much regardless of their accuracy specs.  But for other purposes the difference would be critical. 

I agree with rhb and others that the assumption of gaussian noise should be evaluated, but it is much more critical to evaluate the uncorrelated assumption that goes into RSS calculations.  It is quite possible to have correlated gaussian "noise".  For example take your lab gaussian noise generator and feed it into both UUTs.  The extension into less contrived situations is obvious and includes such things as lab temp, line noise and many other common topics.  Lab temp in many cases is a near gaussian distribution.  Line noise often has a gaussian component along with other often higher amplitude components.

[rhb]

Until you have about 100,000 samples, a known, provably Gaussian distribution will *NOT* look Gaussian.

If you have tens of millions of samples, anything which does conform to the underlying distribution sticks out like a sore thumb.  In multiparameter datasets, it's fairly simple to identify correlated artifacts and if you have a *LOT* of data, you can simply throw anything that looks suspect away.

An extremely useful way to do this is to make 3D histograms as a movie of 2D slices.

I have an intense dislike for "statistics" because of the widespread practice of sprinkling Gauss water on a few hundred samples and making pronouncements without ever addressing the question of whether the Gaussian assumption is valid.

Probability analysis I like.  It does not make any assumptions about the underlying process.  It simply says the frequency  of occurrences of a value.


Dealing with correlated noise requires careful experiment design.  In seismic we filter data based on whether it is or is not correlated.  Sometimes it's the noise that is correlated and sometimes it's the data.  And it commonly involves doing an integral transform, i.e. Fourier, Radon, etc and resorting the data several times.

Reg

[FransW]

Just remember that there is a difference between accuracy and precision.

The only check is to verify and validate the outcome with the "true value".
That is where the real problem is located: absolute measurements do not exist.
They are physically impossible.

That is why calibration labs exist. That is why traceability is the foundation of the
current technology.

https://en.wikipedia.org/wiki/Accuracy_and_precision

Frans

[FransW]

As for seismic data sets: there are numerous ways that the chosen parameters for a survey contribute to the measured results to be interpreted.
The optimal choices are based on human experience with the associated uncertainties.
Exotic technologies (transforms, auto- and crosscorrelation) do not necessarily contribute
to the required accuracy, helas.

However, observed "Gaussian distributions" can be tested for being Gaussian or otherwise.
It will be difficult to explain the match results.

[SilverSolder]

Are we saying that if we only have a small number of measurements,  calling the distribution "Gaussian" is as good as anything else - unless there is some prior knowledge about the process that generates the data, which would make us possibly assign a different distribution?

[CatalinaWOW]

Are we saying that if we only have a small number of measurements,  calling the distribution "Gaussian" is as good as anything else - unless there is some prior knowledge about the process that generates the data, which would make us possibly assign a different distribution?

That is one way to interpret it.  If you want to chase the vortex further you can use statistics to estimate the probability that each of the candidate distributions is consistent with the data, and pick the highest estimate.  But you should again consider the use you are putting this to.  There is a region of high probability of occurance for all distributions.  For a great many of them that region overlaps and the answers don't vary widely in that region.  In some sense, and for some purposes it doesn't matter which distribution you use.  The answers will be similar.  For other purposes it would be wildly wrong.

If your application depends on the details the only answer is more data.  Enough to make a choice between distributions.

[rhb]

Very well put.  The analyst's job is to figure whether it matters or not.  With limited data that can be *very* challenging and if the sample size is small may be impossible.

Good experiment design is absolutely critical. In a seismic survey  one has to make sure it is adequately sampled in X, Y, Z & T.  On a $10 million  3D survey the technical specifications in the contract run for many pages and there is a company rep on board who is constantly checking that everything is within spec.  With the tremendous increase in low cost compute power they will process the data on board to check the results.  No one would use that for anything but field QC.  But because the SNR is so much less than 1, unless the data is run through the full processing sequence there is no way to tell if you have usable data at 10 seconds or if it is all noise.

As I understand the terms (I dealt with the data, not the acquisition), if the data does not meet spec, all ship and crew time are on the seismic company's nickel until the data meets spec.  There are pages of stuff in the contract about how near other vessels can be, marine life, sea state and more strange seeming, but critical specs.  I have only seen one contract and that only for a few minutes.  But I was stunned by how strict the requirements were.   And even more stunned when I realized that the bad data I often saw had met the spec, but still needed a *lot* of work to clean up.

Reg