The "double difference method" - any pointers?

[Cerebus]

I was reading a voltage reference related paper the other day that included a casual, throw away, reference to the "double difference method" in the specific context of using said method to unambiguously identify which of a collection of voltage references was producing noisy or otherwise anomalous data. All in the further context of not having an absolute reference to work from. i.e. They were developing voltage references that were, in some fashion, beyond the references they already had available.

I've tried searching for a "double difference method" that could be the one in question but have come to a dead end. I've encountered mentions of using a similarly named method for finding the epicentre of earthquakes and in the social sciences - both sources of highly noisy data - but I haven't found an unambiguous enough description to either know if I've found the right thing or to apply the method from that discipline to the original question.

Has anyone got useful pointers to a clear description of this method - either specifically in the context of "which voltage reference is lying" or more generally as a statistical method of unambiguously identifying outliers in noisy data?

The volt nuts home-brewing  voltage references can hopefully immediately see how interesting and useful this could be.

Ian

[AndyC_772]

I guess that if you have three references, called A, B and C, and measure each of them in turn against one of the others, then you might get results along the lines of:

A vs B = noisy
A vs C = quiet
B vs C = noisy

In this case, A and C are both quiet, so the comparison between the two yields a quiet result too. But, since B is noisy, measuring it with respect to either of the two others yields a noisy result.

I'm not sure whether that type of comparison would actually warrant a name as such, but it seems like the obvious way to identify which is the outlier. Moreover, it seems fairly apparent that if the three references were 'quiet', 'noisy' and 'noisier', then the set of comparisons between them would show different degrees of noisiness, from which the relative merit of each reference could be readily determined.

[Cerebus]

I guess that if you have three references, called A, B and C, and measure each of them in turn against one of the others, then you might get results along the lines of:

A vs B = noisy
A vs C = quiet
B vs C = noisy

In this case, A and C are both quiet, so the comparison between the two yields a quiet result too. But, since B is noisy, measuring it with respect to either of the two others yields a noisy result.

I'm not sure whether that type of comparison would actually warrant a name as such, but it seems like the obvious way to identify which is the outlier. Moreover, it seems fairly apparent that if the three references were 'quiet', 'noisy' and 'noisier', then the set of comparisons between them would show different degrees of noisiness, from which the relative merit of each reference could be readily determined.

The problem is that all voltage references are noisy and drift, the good ones and the bad ones. So, at any moment a good voltage reference can have wandered off positive from the 'right' value for a bit, another good reference can have wandered off negative and the crappy bad one just happens at that moment to have the 'right' value - how do you tell which is which. The answer is, obviously, long term statistics. But how do you tell - and here comes the magic word - unambiguously in the short term?

I found the paper in question again. Here's the throwaway reference:

(i) The group of eight 10V standards contain three portable units which are calibrated against standards traceable to the National standard of voltage about every nine months. Four of the group of 10 V standards, selected as being those of lowest noise, are intercompared continuously by the double-difference method. This allows any noise to be unambiguously traced to the unit causing it.

The hint I have, from the other uses I found, is that 'difference' here is used in the same sense as 'differential'.

[Cerebus]

I don't know if that's what they mean by "double differential" measurement, but that *IS* the proper way to measure two voltage standards against each other.  You simply repeat this measurement so that each voltage standard you have is measured against each other [N*(N-1)/2 measurements for 'N' voltage standards].

No, it's definitely "double difference" in the quote; and the apparently related statistical material I could hunt down (seismology, social science) is what provided the hint of "differential" but that's in the "derivative" sense (e.g. dv/dt or d²v/dt²) not the "differential pair" or "differential measurement" sense.

Hopefully someone will come along soon who recognises the term. I'm fairly confident it won't be a social scientist but there might be a seismologist kicking around.

[quantumvolt]

I guess that if you have three references, called A, B and C, and measure each of them in turn against one of the others, then you might get results along the lines of:

A vs B = noisy
A vs C = quiet
B vs C = noisy

In this case, A and C are both quiet, so the comparison between the two yields a quiet result too. But, since B is noisy, measuring it with respect to either of the two others yields a noisy result.

I'm not sure whether that type of comparison would actually warrant a name as such, but it seems like the obvious way to identify which is the outlier. Moreover, it seems fairly apparent that if the three references were 'quiet', 'noisy' and 'noisier', then the set of comparisons between them would show different degrees of noisiness, from which the relative merit of each reference could be readily determined.

The problem is that all voltage references are noisy and drift, the good ones and the bad ones. So, at any moment a good voltage reference can have wandered off positive from the 'right' value for a bit, another good reference can have wandered off negative and the crappy bad one just happens at that moment to have the 'right' value - how do you tell which is which. The answer is, obviously, long term statistics. But how do you tell - and here comes the magic word - unambiguously in the short term?

I found the paper in question again. Here's the throwaway reference:

(i) The group of eight 10V standards contain three portable units which are calibrated against standards traceable to the National standard of voltage about every nine months. Four of the group of 10 V standards, selected as being those of lowest noise, are intercompared continuously by the double-difference method. This allows any noise to be unambiguously traced to the unit causing it.

The hint I have, from the other uses I found, is that 'difference' here is used in the same sense as 'differential'.

As you quote some 'scientific' paper here - wouldn't it be scientific of you to name the source. Could help you and be useful for others ...

Anyway, here's your stuff https://en.wikipedia.org/wiki/Difference_in_differences


This qoute from http://web.worldbank.org/WBSITE/EXTERNAL/TOPICS/EXTPOVERTY/EXTISPMA/0,,contentMDK:20188244~menuPK:384339~pagePK:148956~piPK:216618~theSitePK:384329,00.html#difference gives a hint on how to use the method:

'Double difference or difference-in-differences methods
An estimation method to be used with both experimental and non-experimental design

Double difference or difference-in-differences methods compare a treatment and a comparison group (first difference) before and after the intervention (second difference). This method can be applied in both experimental and quasi-experimental designs and requires baseline and follow-up data from the same treatment and control group.

A baseline survey is conducted for the outcome indicators for an untreated comparison group as well as the treatment group before the intervention followed by a follow-up survey of the same sampled observations as the baseline survey after the intervention. If the sampled observations tend to differ in the follow-up survey from the baseline survey, then they should be from the same geographic clusters or strata in terms of some other variable.

The mean difference between the “after” and “before” values of the outcome indicators for each of the treatment and comparison groups is calculated followed by the difference between these two mean differences. The second difference (that is, the difference in difference) is the estimate of the impact of the program (A special case of double differences is “reflexive comparison” that only compares the treatment group before and after the intervention).'

[awallin]

Hopefully someone will come along soon who recognises the term. I'm fairly confident it won't be a social scientist but there might be a seismologist kicking around.

Try searching for 'three cornered hat' or 'N-cornered hat' - quite common method in oscillator/clock measurements.
It works well if the compared standards are roughly equal in performance and fulfil the other assumptions of the method (i.e. noise is not correlated between the standards and they look like a stationary noise process)
http://www.wriley.com/3-CornHat.htm

[Cerebus]

[snip]

As you quote some 'scientific' paper here - wouldn't it be scientific of you to name the source. Could help you and be useful for others ...

In English, putting 'scientific' in quotes like that implies sarcasm which I hope was not your intent. As there's a Thai flag next to your name I'll make the assumption that you're not a native English speaker and give you the benefit of the doubt. If you are a native English speaker then the attitude isn't helpful to anybody and may I suggest that you go forth...

... wouldn't it be scientific of you to name the source ...

Actually the word would be "rigorous" not "scientific".

I tried to do everybody a favour and actually searched all and read most of the whole Metrology category before diving in with questions. The paper is Spreadbury which appears well known enough amongst the cognoscenti here and hence isn't in itself worth re-visiting per se. Quoting a reference would just have been so much extra clutter to detract from the actual question.

Anyway, here's your stuff https://en.wikipedia.org/wiki/Difference_in_differences


This qoute from http://web.worldbank.org/WBSITE/EXTERNAL/TOPICS/EXTPOVERTY/EXTISPMA/0,,contentMDK:20188244~menuPK:384339~pagePK:148956~piPK:216618~theSitePK:384329,00.html#difference gives a hint on how to use the method:

'Double difference or difference-in-differences methods
An estimation method to be used with both experimental and non-experimental design

Double difference or difference-in-differences methods compare a treatment and a comparison group (first difference) before and after the intervention (second difference). This method can be applied in both experimental and quasi-experimental designs and requires baseline and follow-up data from the same treatment and control group.

A baseline survey is conducted for the outcome indicators for an untreated comparison group as well as the treatment group before the intervention followed by a follow-up survey of the same sampled observations as the baseline survey after the intervention. If the sampled observations tend to differ in the follow-up survey from the baseline survey, then they should be from the same geographic clusters or strata in terms of some other variable.

The mean difference between the “after” and “before” values of the outcome indicators for each of the treatment and comparison groups is calculated followed by the difference between these two mean differences. The second difference (that is, the difference in difference) is the estimate of the impact of the program (A special case of double differences is “reflexive comparison” that only compares the treatment group before and after the intervention).'

That's exactly representative of the kind of thing I found that didn't quite jibe with the original context. There's no "intervention" in our case to compare for and I can't quite seem to form a hypothetical "intervention" (like "God just make reference 3 especially noisy") to form a hypothesis from that fits the method. Or if that is the right kind of hypothesis I can't just quite see how to apply it here.

An aside:

Statistics, as of itself, doesn't make my head hurt but most people's attempts to teach it, explain it or explain how to apply it do. As the bits I do understand seem quite straight forward and simple - for example, the central limit theorem - I have a strong suspicion that most of the people I've got this feeling from don't really understand it properly themselves. The worrying thing about this feeling of suspicion is that I get it from the people who most need to properly understand the subject - University lecturers tasked with teaching it, medical researchers, social scientists, polititical scientists etc. etc.

To keep quantumvolt happy, herewith a complete bibtex citation for the paper in question:

[Cerebus]

Hopefully someone will come along soon who recognises the term. I'm fairly confident it won't be a social scientist but there might be a seismologist kicking around.

Try searching for 'three cornered hat' or 'N-cornered hat' - quite common method in oscillator/clock measurements.
It works well if the compared standards are roughly equal in performance and fulfil the other assumptions of the method (i.e. noise is not correlated between the standards and they look like a stationary noise process)
http://www.wriley.com/3-CornHat.htm

I don't know if that's what the original author [Spreadbury] was referring to by another name but it certainly makes a lot sense in this context and looks like a useful approach to solving the problem. I have some suspicion that it is the same and that Spreadbury was playing fast and loose with the terms difference and variance.

It's the throw-away use of the term "double difference method" without a citation or further explanation that lead me to believe that this was a well known method that I just, for whatever reason, had never heard of before.

Thanks, that was helpful.

[alanambrose]

I'm also interested in n-cornered hat re volt refs - I think this might be key to characterising drift.

A.

[Cerebus]

I think I've got my answer, at least for practical purposes, in the three-cornered hat method. I'll probably come back and revisit this here in a while (think months not weeks - I need to build the stuff I want to measure first) with some working code that I'll try and keep as general as possible.

Thanks for everyone's comments.

Ian

[alanambrose]

Ah, I just noticed, there's some good discussion here which talks about comparing references:

https://www.eevblog.com/forum/testgear/fluke-732b-repair/25/

A.