Measuring DUT low frequency noise 0.01-100 Hz Below noise floor of the test equi

[rhb]

Temperature, supply voltage changes (mains voltages vary +/- 5%) are correlated noise.  In this case the desired signal is the uncorrelated reference noise.

I described the basic process of separating correlated and uncorrelated series.  Which is the signal and which is a distinction made by the user.  What I described is standard practice and one of the first things taught in DSP 101.

I was and still am  astonished by the consistent refusal to provide data.  I was very accustomed to not having access to the raw data in the oil industry, but there are valid commercial reasons for that.  I am forced to conclude that my mathematical background is threatening.  Can't grasp why someone would feel threatened by my attempting to teach them math.  If anyone has an explanation I'd love to hear it.

@andreas posted 4 aging curves with the remark "try to find the exponent of these" as two of the curves were quite flat.  I was ready to take him up on it and asked him to post the data in CSV format.  Never happened.  Andreas says it can't be done and makes sure he can't be proved wrong by not letting me try.

So instead we must wait until I've collected several years of data myself.  But I'll have a 100 channel 3497A & 3458A measuring about 80 Vrefs with the other channels monitoring the supply voltages and temperatures of the Vrefs.

Have Fun!
Reg

[moffy]

A nice simple explanation of using cross correlation for extracting a signal from noise, just for reference purposes: https://www.zhinst.com/others/en/blogs/how-reduce-noise-floor-and-improve-snr-employing-cross-correlation-techniques
also: https://www.sciencedirect.com/science/article/pii/S0263224121008095

[Gerhard_dk4xp]

<     https://tf.nist.gov/general/pdf/1133.pdf    >


<     https://arxiv.org/pdf/1003.0113.pdf   >
Or the author's most interesting web site:

<   https://rubiola.org/      >, for example item 11.

regards, Gerhard

[tomato]

I was and still am  astonished by the consistent refusal to provide data ... I am forced to conclude that my mathematical background is threatening.  Can't grasp why someone would feel threatened by my attempting to teach them math.  If anyone has an explanation I'd love to hear it.

The explanation is likely very simple; people don't offer you any help because they're put off by your attitude.  The fact that you think it's because they're intimidated by your mathematical background is just one example of that attitude.

[rhb]

I was and still am  astonished by the consistent refusal to provide data ... I am forced to conclude that my mathematical background is threatening.  Can't grasp why someone would feel threatened by my attempting to teach them math.  If anyone has an explanation I'd love to hear it.

The explanation is likely very simple; people don't offer you any help because they're put off by your attitude.  The fact that you think it's because they're intimidated by your mathematical background is just one example of that attitude.

My attitude is a consequence, not native to me.  When I came along asking for data it was refused.  After many years, I still don't understand why trying to teach someone some very cool math is offensive.  So I'm guessing.

Ketchup!

Edit: I just realized that from the number of posts and where they appear you are almost certainly someone's sock puppet.  Question is, whose?

[tomato]

I was and still am  astonished by the consistent refusal to provide data ... I am forced to conclude that my mathematical background is threatening.  Can't grasp why someone would feel threatened by my attempting to teach them math.  If anyone has an explanation I'd love to hear it.

The explanation is likely very simple; people don't offer you any help because they're put off by your attitude.  The fact that you think it's because they're intimidated by your mathematical background is just one example of that attitude.

My attitude is a consequence, not native to me.  When I came along asking for data it was refused.  After many years, I still don't understand why trying to teach someone some very cool math is offensive.  So I'm guessing.

Regardless, you still own your attitude.

Edit: I just realized that from the number of posts and where they appear you are almost certainly someone's sock puppet.  Question is, whose?

No, I am not a sock puppet.

[rhb]

Actually, not only am I sure you're a sock puppet, but I  am pretty sure I know whose hand is in the sock and can prove it.  Little thing called "lexical analysis".  Funny thing about humans.  They all have unique signatures which can be identified based on word choice and sentence structure.

I'd very much prefer to be left alone than have you banned.

[tomato]

Actually, not only am I sure you're a sock puppet, but I  am pretty sure I know whose hand is in the sock and can prove it.  Little thing called "lexical analysis".  Funny thing about humans.  They all have unique signatures which can be identified based on word choice and sentence structure.

I'd very much prefer to be left alone than have you banned.

I'm sure a moderator can verify that I'm not someone's sock puppet. 

You asked for an explanation of why people don't share their data with you, and I provided a possible reason.  That's it, end of story. 

[Revky27]

A fascinating process, using the randomness of two channels noise to extract the common input signal via correlation.

P.S. Saving your explanation for reference purposes.

Actually, you've got it backwards.  In this case the signal is uncorrelated, but the noise is correlated.  So one exploits the fact that a cross correlation between uncorrelated series is zero.  This allows separating the correlated noise and removing it.

The OP wants to measure low frequency voltage reference noise.  All those simultaneous deviations of the reference graphs so popular among the volt-nuts.  I have repeatedly asked for ASCII data so I could demonstrate the process, but for some reason volt-nuts don't like to share data in usable form.  Same for time-nuts.  As much verbiage as has been posted about thermal stability and other noise processes, etc, one would think they'd welcome the chance to obtain cleaner data, but you'd be very wrong.  I was sent a plot of 4 frequency references by a time-nut.  It had obvious strong correlated noise at  femptoHz  frequencies and would have been a snap to clean up.  But when I asked for the data he had plotted he 'had to think about it".  I never got any data and dropped off the  time-nuts list.

The only rationale I can imagine is that they are mortally afraid of mathematics.  A successful proof would compel them to learn math.  Lacking confidence in their abilities, they avoid the issue by refusing to allow me to have the data except as pictures which are not usable for mathematical analysis.

Have Fun!
Reg

Sorry but how can you say the 'noise is correlated'. What noise do you have that behaves like this?

[rhb]

Temperature, barometric pressure, humidity, mains variation are all noise processes which are correlated among references in the same enclosing space.  There are likely others, but you have to peel the onion one layer at a time.

Those 4 are first order noise processes. at low frequencies.

Have Fun!
Reg

[MegaVolt]

Temperature, barometric pressure, humidity, mains variation are all noise processes which are correlated among references in the same enclosing space.  There are likely others, but you have to peel the onion one layer at a time.

Those 4 are first order noise processes. at low frequencies.

I watch with great interest your desire to show good mathematics. Unfortunately I don't have much data.

But perhaps you could show an example with numbers using some synthetic data generated artificially. Those. take the signal. Add noise to it. Add temperature and pressure effects. Evaluate the signal parameters using classical methods and the method you propose.

It would be very interesting to see how it was possible to get the signal out of the noise when conventional methods cannot offer anything.

Thanks in advance.

[rhb]

Please read:

https://www.eevblog.com/forum/projects/so-you-want-to-do-digital-signal-processing/msg5284960/#msg5284960

Try doing some basic experiments using Octave.  I've been doing DSP for 42 years.  I don't need to repeat my first 6 weeks of DSP.  Which was on the job reading 4 books evenings and weekends and processing seismic data during the day.

Generate 3 random series, cross correlate them, then add one series two the other two series and cross correlate the pair.  The central value of each of the initial cross correlations will be zero.  The 3rd xcor will not be zero.  The longer your time series, the better you will be able to separate the added series from the other 2 series to which it has been added.

FWIW This *is* the conventional method.  Please read Bendat & Piersol.  They are the masters of the conventional method which was pioneered by Norbert Wiener are MIT.  Sparse L1 pursuits is a step farther.


Have Fun!
Reg

[MegaVolt]

Please read:

https://www.eevblog.com/forum/projects/so-you-want-to-do-digital-signal-processing/msg5284960/#msg5284960

Try doing some basic experiments using Octave.  I've been doing DSP for 42 years.  I don't need to repeat my first 6 weeks of DSP.  Which was on the job reading 4 books evenings and weekends and processing seismic data during the day.

Generate 3 random series, cross correlate them, then add one series two the other two series and cross correlate the pair.  The central value of each of the initial cross correlations will be zero.  The 3rd xcor will not be zero.  The longer your time series, the better you will be able to separate the added series from the other 2 series to which it has been added.

FWIW This *is* the conventional method.  Please read Bendat & Piersol.  They are the masters of the conventional method which was pioneered by Norbert Wiener are MIT.  Sparse L1 pursuits is a step farther.


Have Fun!
Reg

I have basic knowledge of DSP. I have a gap in using cross-correlation. The correlation between two sequences is one number. Let's say I got 0.5.

How will this number help me remove correlated noise from one of the sequences?

[rhb]

You average all the cross correlation to derive an estimate of the correlated noise which you then subtract from each time series.

The longer the series, the better your estimate of the correlated noise.  As the series of random uncorrelated series gets longer the cross correlation approaches zero.  That improves the averaged cross correlations accuracy.

Random Data by Bendat & Piersol is the canonical master treatment of the mathematics of DSP.  I started with the 2nd ed in the late '80s and bought the 3rd & 4th.  It is all pure applied mathematics with equations on almost every page.  But if you really want to understand DSP, it is the "sine quo non" for mastering the topic.

Temperature, pressure and humidity affect the heat flow from the Vrefs.  As it is radiative, conductive and convective the mathematics get very messy.  Carslaw & Jaeger have the basic details in their classic on "Conduction of Heat in Solids".   An empirical solution would be a 6-8 week job, $$$$$.  With real data spanning 10 years I'll do it for free.  Making a numerical example is almost as much work.  I have better uses of my time.

Thanks for asking a sensible question instead of a demand that I do a bunch of work which is of no use to me.

In regard to sparse L1 pursuits,  section 12.4 of "A Wavelet Tour of Signal Processing" 3rd ed will give you the flavor.  Understanding the proofs requires reading Mallat cover to cover and then Foucart & Rauhut.  That took me 3 years to do including the original papers which were about 2/3 of what I read in those 3 years.

My BA is in English lit.  If *I* can master Foucart & Rauhut, anyone can.  But it demands very strong motivation.  The proof of the first of 3 theorems by Donoho took 14 of 15 pages!

NB:  The first order step would be to sum all the series for an initial estimate of the correlated noise and then refine that estimate using the cross spectra.  So my description is what I'd tell the PhD across the hall at work, not a step by step description.  We *all* forget stuff.  I ran "an orphan home for lost problems"  at 5 oil companies.  Most of the time all I did was ask a few questions and the person responded,  "I know how to do that!" and rushed back to their office.


Have Fun!
Reg

[moffy]

So if I can chip in what I think, if I have 3 inputs all going to an ADC or multiple ADCs, if I connect a signal to two of them and zero to the input of the third, the third input would provide the common correlated noise due to environmental factors and can then be used against the two inputs with a signal to help remove just this common environmentally induced noise, or noise common to all channels. Once this noise is removed from the two channels with the signal cross correlation on the two signal channels can be used to uncover the original signal buried in the uncorrelated noise.

[rhb]

That's the cool part.  You can recover the uncorrelated noise as in the Vref case, or you can recover the correlated noise.

It's a signal & noise  separation technique.  Totally agnostic about what is signal and what is noise.

In the seismic world this is done in multiple dimensions with complex spatial operators.

Really recommend Bendat & Piersol.

Have Fun!
Reg