Author Topic: Applications of Sparse L1 Pursuits to Precision Reference  (Read 2871 times)

0 Members and 1 Guest are viewing this topic.

Offline rhb

  • Super Contributor
  • ***
  • Posts: 1603
  • Country: us
Re: Applications of Sparse L1 Pursuits to Precision Reference
« Reply #25 on: April 06, 2018, 10:25:27 pm »
Well, I felt as if I'd been called out in class ;-)

Given:

Aj, Bj, Cj..... Nj  at each round of measurements, calculate all the pair wise products and add to the accumulated correlation coefficient.

Phi(A,B) += Aj*Bj is the running estimate of the crosscorrelation coefficient of A & B.  If I were doing this I would put all the correlation coefficients in a file as ASCII text:

number pair

sort -k 1n <file >sorted

and then plot sorted using gnuplot. In general one gets a plot of the cumulative distribution function.  As most data are quasi-Gaussian this takes on a sigmoidal shape with the outliers at the right and left.

The optimal measurement length is as long as possible, as often as possible.  However, daily over a few years is adequate.  The lower the drift, the longer the series needed to characterize it.

I should like to note that the measurement drift of the DVM is correlated across devices.  That introduces an error in the correlation coefficient estimate.  I'm still contemplating the particulars, but a sparse L1 pursuit should be able to separate the DVM drift from the reference drift if there are multiple references.  The part I've not determined is how the number of references in excess of two affects the accuracy of the DVM drift estimate.  I'm also a bit unclear if it's possible using Wiener's L2 methods.  Likely one can, but the procedure may be rather involved.

If there is interest from TiN, cellularmitosis, Andreas and others who are measuring and comparing multiple references I'll write a bespoke program to do the calculations and generate files for use with gnuplot.

I know little about the Allan deviation as I have never coded it, just looked at plots of clocks.  So I'll need to read the definition closely and work out the arithmetic required.  The article I checked suggested it was far simpler than I thought.

If you are at all interested in problems such as posed by Zia, you really should get a copy of "Random Data" by Bendat and Piersol.  It is very readable and describes in clear language how to solve most problems.  There are a few things that sparse L1 pursuits can do that can't be done with classical L2 methods, but pursuits are not a replacement for classical methods. Allan Piersol passed away, so the 4th ed is the last.  Octave/MATLAB will handle all the numerical chores.
 

Offline zhtoor

  • Frequent Contributor
  • **
  • Banned!
  • Posts: 337
  • Country: pk
Re: Applications of Sparse L1 Pursuits to Precision Reference
« Reply #26 on: April 06, 2018, 10:43:36 pm »
I should like to note that the measurement drift of the DVM is correlated across devices.  That introduces an error in the correlation coefficient estimate.

this might help.

http://rubiola.org/pdf-articles/archives/2010-arxiv-1003.0113v1-xspectrum.pdf

best regards.

-zia
 
The following users thanked this post: rhb

Offline zhtoor

  • Frequent Contributor
  • **
  • Banned!
  • Posts: 337
  • Country: pk
Re: Applications of Sparse L1 Pursuits to Precision Reference
« Reply #27 on: April 07, 2018, 08:46:46 pm »
hello,

refer to test bench #1 as posited above (reply #19):-

1. for a given k <= floor(N/2) select a random subset of k VREFS(1..N)
2. select the switch group A to this subset.
3. select the switch group B to a random subset of remaining VREFS. (N maybe odd)
4. do a measurement / analysis on lines VA, VB.
5. repeat 1..4

now:-

1. can we detect an outlier and in how many measurement cycles ?
2. can we draw a curve of the number of measurements required to achieve (1) vs. N/k, what form would it be ?

best regards.

-zia
 

Offline rhb

  • Super Contributor
  • ***
  • Posts: 1603
  • Country: us
Re: Applications of Sparse L1 Pursuits to Precision Reference
« Reply #28 on: April 07, 2018, 10:26:28 pm »
I'm going to restate the problem:

Given N references and two DVMs, how can the worst references be found in the least time?

divide the references in two groups of approximately equal size

compute the Allan deviation of the two groups

the group with the largest deviation contains an outlier

divide the largest deviation group into two groups and repeat until the device with the largest drift is found

This is search by bisection which is guaranteed to converge in P steps for 2**P <= N.  The length of time it takes depends upon how large the drift of the outlier is.  Each step in the search must run long enough to produce a robust estimate of the variance.  In term of wall clock time to find the best and worst references measuring them individually and computing the cross spectra is the best approach.  Finding the next worst reference would require completing the search again with the worst reference removed from the pool.

Which brings up the problem of multiplexing the input to a DVM.  What's the best way to do this?  FET switches/multiplexers?  Relays?  What type of relays?

I'd like to read N references and N*(N-1) differentials with my 3478A.  In my case N=3, at least at present.  I tried a search of the forum with google, but didn't find anything beyond buying commercial units and it's hard even with commercial units.

As an aside, "Numerical Recipes" includes a number of "optimal" searches.  But they are not guaranteed to be faster than bisection and can have search times which are significantly worse.  I have never understood why anyone would use them unless they know with certainty  a priori that the worst cases cannot arise. I was in grad school when the first edition came out and wildly enthusiastic.  I bought the 2nd, but I haven't looked at either in many years and can't think of a reason I would.  I have *never* used anything but bisection in my professional work.  It is guaranteed to be reliable if you code it correctly.  But it was several years between the first publication of a bisection algorithm and a correct bisection algorithm.  It's maddenlngly easy to get the test edge cases wrong.

 

Offline zhtoor

  • Frequent Contributor
  • **
  • Banned!
  • Posts: 337
  • Country: pk
Re: Applications of Sparse L1 Pursuits to Precision Reference
« Reply #29 on: April 07, 2018, 10:39:03 pm »
Which brings up the problem of multiplexing the input to a DVM.  What's the best way to do this?  FET switches/multiplexers?  Relays?  What type of relays?

have a look at this:-

http://www.eevblog.com/forum/metrology/low-thermal-emf-scanner-and-ordinary-non-latching-relays/msg1064900/#msg1064900

best regards.

-zia
 

Offline zhtoor

  • Frequent Contributor
  • **
  • Banned!
  • Posts: 337
  • Country: pk
Re: Applications of Sparse L1 Pursuits to Precision Reference
« Reply #30 on: April 07, 2018, 10:41:11 pm »
I'm going to restate the problem:

Given N references and two DVMs, how can the worst references be found in the least time?

how about restating the problem as a compressed sensing problem?

best regards.

-zia
 

Offline rhb

  • Super Contributor
  • ***
  • Posts: 1603
  • Country: us
Re: Applications of Sparse L1 Pursuits to Precision Reference
« Reply #31 on: April 07, 2018, 11:09:01 pm »
I'm going to restate the problem:

Given N references and two DVMs, how can the worst references be found in the least time?

how about restating the problem as a compressed sensing problem?



Sparse L1 pursuits are a method of solution that allows solving problems traditional L2 methods can't handle.  Amazing as it it is, it doesn't perform miracles.

If we knew with high certainty the functional form of the drift curve and we had measurements from initial startup, it *might* be possible to solve for the individual drift curves from a measurement of the sum.  But it seems unlikely.  Given the above information it would take a week working full time to determine if it was possible at all.  And then even with the answer, which unit is which?

I was just pricing a 3457A w/ 44492A.  The price made me ask why am I doing this?  Building a multiplexer seems more attractive. It's creation rather than consumption.
 

Offline rhb

  • Super Contributor
  • ***
  • Posts: 1603
  • Country: us
Re: Applications of Sparse L1 Pursuits to Precision Reference
« Reply #32 on: April 07, 2018, 11:23:50 pm »
Which brings up the problem of multiplexing the input to a DVM.  What's the best way to do this?  FET switches/multiplexers?  Relays?  What type of relays?

have a look at this:-

http://www.eevblog.com/forum/metrology/low-thermal-emf-scanner-and-ordinary-non-latching-relays/msg1064900/#msg1064900

best regards.

-zia


That's very interesting.  It appears that it would be pretty easy to correct. Also rather annoying that google didn't find it.
 

Offline zhtoor

  • Frequent Contributor
  • **
  • Banned!
  • Posts: 337
  • Country: pk
Re: Applications of Sparse L1 Pursuits to Precision Reference
« Reply #33 on: April 08, 2018, 05:09:37 am »
I was just pricing a 3457A w/ 44492A.  The price made me ask why am I doing this?  Building a multiplexer seems more attractive. It's creation rather than consumption.

hello,

the hp3456a may be a better and a cheaper choice. a number of pro's on this board swear by it.

best regards.

-zia
 

Offline rhb

  • Super Contributor
  • ***
  • Posts: 1603
  • Country: us
Re: Applications of Sparse L1 Pursuits to Precision Reference
« Reply #34 on: April 08, 2018, 05:54:49 am »
The 3457A has a a couple of different multiplexer modules the 44491A & 44492A.  The 3456A doesn't.  But the cheapest I could find was a 3457A w/ 44492A for $500 which is pushing what I want to spend on this.  A 44492A alone was typically about $250-300.

Based on the information in this thread:

http://www.eevblog.com/forum/metrology/low-thermal-emf-scanner-and-ordinary-non-latching-relays/msg1064900/#msg1064900

I shouldn't have much trouble building a  10-12 channel scanner and simply correcting for the thermal effects.  The biggest obstacle is my lack of PCB design tool experience.  I tried doing some simple stuff a few years ago and got really frustrated trying to sort out how to do what I wanted.  I never completed a board design.  This is not something I would build without a full ground plane and guarding.  So unless someone volunteers to design a board, it won't happen soon.

I put a small loop on a scope probe and was able to track down SMPS noise from my PC which I was able to squelch with a piece of hardware cloth, but I'm now trying to figure out the path for the SMPS noise from the Instek MSO-2204EA.  Somehow it is coupling into the probe on the power feed to the basic-lm399 boards.  The power lead is now shielded PTFE twisted pair with the shield connected only at the PSU end.  I have an assortment of clamp on ferrite chokes on order.

Screening the PC PSU was so successful that I'm going to pull the light fixture down and examine the screen bonding and the integrity of the mains ground lead.  It was difficult working on a ladder over my head.

 

Offline Kleinstein

  • Super Contributor
  • ***
  • Posts: 4552
  • Country: de
Re: Applications of Sparse L1 Pursuits to Precision Reference
« Reply #35 on: April 08, 2018, 06:10:11 am »
The 3457 is not that bad, though a bid noisy and has odd ranges ( 3 V and 30 V).


PCB design can be quite time consuming, especially with more than 1 layer.

For a scanner it depends on the requirements.  Mechanical relays usually have better isolation, but may not be 100% reproducible in the sub ┬ÁV range. If some leakage can be tolerated CMOS switches are a really easy choice. The DG508 can be a simple way that even includes some protection.  JFET switching can get tricky to get the drive levels - one may need some kind of guard amplifier to get the right level to drive the gates.  With already a scanner card in the 3457 one may not need an extra scanner.
 

Offline rhb

  • Super Contributor
  • ***
  • Posts: 1603
  • Country: us
Re: Applications of Sparse L1 Pursuits to Precision Reference
« Reply #36 on: April 08, 2018, 08:39:20 am »
R-on of ~175 ohms  seems to make the DG508B unattractive for measuring voltage references unless there's a way to null it out.   My 4.5 digit Tenma DMM reads 0.50 uA  connected in series with the leads from an LM399 to my 3478A which shows a  300 uV drop  from the shunt reisstor in the Tenma.

The tempco of the DG508B from the datasheet is about  0.76 ohms/C which translates to 0.38 uV/C and a nominal offset voltage drop of 85 uV feeding the 3478A.  Of greater concern was the lack of any indication of aging performance,

I've been chasing down EMI in my work area much of the day, so my brain is elsewhere.  Do you know of a circuit for using the DG508/9A for ppm measurements?  The relay behavior that @zhtoor pointed to seems to me easier to model and correct.

My big interest in metrology at the moment is sparse L1 pursuits.  And I have the continual annoyance of the UIs on my Instek scopes as motivation to get back to learning to make a Zynq dance to my tune.  I'm getting very tempted to get a 2nd 3478A and build a 12 input relay scanner if someone will help out with the scanner PCB.  I feel very confident I can measure and correct the thermal EMF shown in the thread @zhtoor cited.

I want to *do* something, not buy something.  I invested a lot of time learning about sparse L1 pursuits.  I'd like to use it to do something original.  Personal conflicts with my supervisor at Austin led to my not getting my doctorate. I don't care about the certificate, but I am unhappy about not being allowed to complete the project.  It was good work, but was done a few years later by someone else.

I hope no one will take umbrage at my mentioning this here,  but I connected a $2.99 HF DMM to one of the LM399s today and trimmed it to match the 3478A.  Amazingly, it was within 0.5% when I tested it.  Rather  fiddly to adjust, but I got it to match the first 3 digits of the 3478A. .  I have no idea how old it is other than they have sold 2 newer models since I bought this one.






 

Offline cellularmitosis

  • Supporter
  • ****
  • Posts: 1030
  • Country: us
Re: Applications of Sparse L1 Pursuits to Precision Reference
« Reply #37 on: April 08, 2018, 03:07:01 pm »
Hey Reg,

I would definitely encourage the DIY scanner route over the $500 used HP gear route.  $500 buys a lot of latching relays, and I think doktor pyta just recently reported that his DIY design had a thermal error of something like 10nV.  Pretty amazing for a budget of only a few bucks per channel.

(I'm actually working on a little DIY LM399 scanner myself, but it might be a few weeks before I have time to work out a board design)

If you were constrained by not being able to design a PCB, you could pick up one of the cards which goes into those scanners, and use an Arduino to drive it directly.  For example, searching for "03497-66509" (on ebay) shows a number of listings for under $50.

Either way, a fun little project for sure!

edit: link to doktor pyta's result: http://www.eevblog.com/forum/metrology/measuring-16-signals-with-one-multimeter/msg1463391/#msg1463391

edit2: looks like he is using G6KU-2F-Y relays, which are under $5 https://www.digikey.com/product-detail/en/omron-electronics-inc-emc-div/G6KU-2F-Y-TR-DC5/Z3706CT-ND/3908609
« Last Edit: April 08, 2018, 03:22:28 pm by cellularmitosis »
LTZs: KX FX MX CX PX Frank A9 QX
 

Offline rhb

  • Super Contributor
  • ***
  • Posts: 1603
  • Country: us
Re: Applications of Sparse L1 Pursuits to Precision Reference
« Reply #38 on: April 08, 2018, 10:15:26 pm »
Thanks for the tip on the boards.  They're reasonably priced and the critical analog part is done to the best professional standards.  I just agreed to buy an HP 8569A.  So I think it's probably time to slow down buying equipment

I'd still like to try correcting the thermal errors in non-latchng  relays.
 

Offline rhb

  • Super Contributor
  • ***
  • Posts: 1603
  • Country: us
An initial Vref system model
« Reply #39 on: April 08, 2018, 11:36:11 pm »

This is my candidate  system model:

initial voltage
1/f noise
thermal Johnson noise
thermal errors
aging drift

This is my candidate equation:

Vref(f) = V_initial + 1/f_noise(f) + Johnson_noise(f) + thermal_errors(f) + aging_drift(f) + mains_noise(f)

The 1/f noise has curvature at the LF end of the spectrum and the Johnson noise should be flat until it reaches the anti-alias filter Fc,.  The thermal errors  correlate with temperature.  The aging drift has curvature early in the series which flattens out over time, so I'll have to window the data into segments to derive that term.  The mains noise is narrowband, so it should be easy to model.

The early aging data is critical to accurately modeling aging.  Once the drift curve starts to flatten out it becomes increasingly difficult to separate from the 1/f noise.

PLC != 0 applies a sinc(f) filter to the data.  This has the consequence of making it difficult to separate the Johnson noise from the 1/f noise because the full amplitude frequency range is very narrow.  It remains to be seen how severe an impact it has on the separation.
 

Offline rhb

  • Super Contributor
  • ***
  • Posts: 1603
  • Country: us
A first order voltage reference measurement approximation
« Reply #40 on: April 15, 2018, 11:08:44 pm »
After reading various descriptions of the 3458A I think it wise to skip over the multislope operation for the initial model.  My current model candidate is as follows:

1) Generate the noise component limited to one 1/f noise source and one thermal noise source

          compute Vnoise = Gaussian(Tmean, Tsigma) + A*Poisson/f in the frequency domain from DC to 3 KHz  to create a one hour record using     
         four independent Mersenne Twister PRNG sequences for the 1/f and thermal noise, one for the real part and one for the imaginary part.   
         The frequency sampling is 278 uHz.

2) inverse transform to time and add Vdc and mains noise

3) integrate for PLC=10 and subsample to 6 Samples/S

This basically assumes a "perfect" 3458A.  So all the noise is attributed to the reference which is assumed to have no drift over the course of one hour.

The Nyquist is 3 Hz, but the first notch is at 6 Hz, so everything above 3 Hz is aliased.  This seems to me the easiest way to reproduce the effect of the integration in the 3458A.

The experiment is to see how much data I need to recover Vdc to  < 1 ppm if I account for the noise explicitly in an L1 inverse problem.  This is not a sparse L1 pursuit, that implies multiple models for the 1/f noise and trying to find the one that fits best. Or more than one noise source with different characteristics.  It's the first step towards doing that and it seems that no one has done this. based on what I found searching IEEE Explore.

What I have in mind doing was not practical when the 3458A was designed.  I was in grad school in Austin at the time.  We could barely do L2 problems of this size on an 11/780.   I don't even want to think about how long a 22 million sample FFT would have taken.  Probably a couple of days.   Now it's a trivial problem.

[copied from separate thread which I should not have started]

This is the first step in testing the model in the previous post.  Once I have verified this can be solved I'll add the other errors to the model.

I had been concerned about aliasing, but as thermal noise is uncorrelated  one gets an interesting filter effect.  The higher order aliases cause the noise to increasingly cancel as the frequency approaches the Nyquist of 3 Hz.

I'll need to evaluate the expected amplitude of the thermal noise and the 1/f noise and study the cross correlation to get an idea of the limitations on separating Vdc from the noise.  The driving concept is that we can estimate the magnitude of the error at DC from the shape and amplitude of the noise components at higher frequencies and use that to correct the estimate of Vdc.

 

Offline RandallMcRee

  • Regular Contributor
  • *
  • Posts: 226
  • Country: us
Re: Applications of Sparse L1 Pursuits to Precision Reference
« Reply #41 on: April 16, 2018, 12:48:37 am »
I created a java program to model voltage references a while ago.

https://github.com/RMcRee/ModelVoltageReference

It models pink noise and temperature drift. It does use Mersenne Twister, FYI.

Maybe it will be useful for you. The idea there was to recover individual reference behaviour by measuring the differences between each reference to one another. Presuming a known starting point, of course. Used LSQ to impute the actual voltages.

Randall
 

Offline rhb

  • Super Contributor
  • ***
  • Posts: 1603
  • Country: us
Re: Applications of Sparse L1 Pursuits to Precision Reference
« Reply #42 on: April 16, 2018, 01:51:06 am »
What were your results?  Did you take it any farther?  Tell me more.
 

Offline RandallMcRee

  • Regular Contributor
  • *
  • Posts: 226
  • Country: us
Re: Applications of Sparse L1 Pursuits to Precision Reference
« Reply #43 on: April 16, 2018, 02:32:34 am »

So, yes, if all that happens to your vrefs is related to temperature you can track a set of references together.

What discourages me is that there is so much unaccounted for: long term drift, popcorn noise, thermal lag. This is not to mention the corresponding problems in the measurement system itself. Although this is presumably mitigated somewhat by measuring voltage differences.

The problem with my setup is that, once lost, the LSQ cannot recover the actual state of the whole system without redo-ing the initial (assumed) precise, calibrated measurement. That defeats the purpose since, if you had the ability to calibrate at any time, why bother with the LSQ ensemble? LSQ always comes up with a solution and you have no way of knowing if things have gone awry due to one of the unknown, not-accounted-for factors above.

All in all, I think it may be possible for you to achieve some of your goals with your more sophisticated techniques. I can't say my experiment proves it one way or another. But perhaps it does give you some things to think about.

Sort of on a side note. Beginners might wonder why the industry produces so many LT6655 type voltage references that have comparatively low initial accuracy and 2ppm/C temp change. Most ADCs for example have built-in references with those characteristics or worse. The answer is that they are suitable for ratiometric measurements. For metrology the bar is higher. Just thought I would throw that in since I have not seen this written down anywhere. Now, I'm a beginner, too, so if any of this is wrong or misleading, wait a few hours for the corrections to pour in!

 
The following users thanked this post: TiN

Offline rhb

  • Super Contributor
  • ***
  • Posts: 1603
  • Country: us
Re: Applications of Sparse L1 Pursuits to Precision Reference
« Reply #44 on: April 16, 2018, 04:02:24 am »
Thanks.  I don't have a solution for the lack of an initial reference.  I'm afraid that is essential.  All the rest I *think* can be handled.  That's why I have to try this out.

I've been working on the effect of thermal (aka Johnson) noise in an integrating ADC.  I'm solving for the amplitude of the aliased noise after integration over a fixed window.  As it is Gaussian, zero mean noise, summing the aliases cancels by 1/sqrt(n).  So it seems to me that the amplitude of the thermal noise after aliasing and integration is:

J(f) = sinc(f) - sinc(f-pi)
J(f) += sqrt(2)*(sinc(f-pi) - sinc(f+pi))
J(f) += sqrt(3)*(sinc(f+pi) - sinc(f-2*pi))
J(f) += sqrt(4)*(sinc(f-2*pi)-sinc(f+2*pi))
.
.
.


so you have the sum of one noise process, two processes, three, etc. In addition to the noise level of each alias getting smaller, the difference of the terms is getting smaller.

the model is a sinc(f) convoived with a spike series in f. The terms converge rapidly to zero as j increases. So at any frequency you have all of the positive and negative frequency aliases summed together.  I've never dealt with anything like this before.  I'd be grateful for input from someone else with a DSP background.  I don't have a Stanford PhD across the hall to talk to anymore. The wall only responds when I walk into it.  Talking to it does no good and walking into it hurts.

I've attached a plot of the alias envelope amplitudes.  The aliases of concern  are to the left of Pi/2.

[edit; typo in plot.  fixed expression ]
« Last Edit: April 16, 2018, 09:27:28 am by rhb »
 

Offline rhb

  • Super Contributor
  • ***
  • Posts: 1603
  • Country: us
Quite a surprise!
« Reply #45 on: April 17, 2018, 02:17:02 am »
Yesterday I couldn't figure out an algorithm for calculating the amplitude of thermal noise from an integrating ADC.  This morning it was easy.

It turns out I was quite wrong about convergence.  It is very slow.

The plots have the frequency normalized to the sample rate. So 0.5 is Nyquist.

Whereas 1/f noise decreases with frequency, the thermal noise increases quite a bit.  Much more than I expected.  The opposite slopes  bode well for separating the thermal and 1/f noise from the estimated DC value.  The statistical meaning of Vdc after accounting for the noise is yet to be determined.

The plots should be self explanatory.  Nyquist is in the middle.  Not quite sure what to call the right hand half from 0.5 to 1.0.  That is not in the spectrum of the data.

C code is attached if any want to play.
 

Offline rhb

  • Super Contributor
  • ***
  • Posts: 1603
  • Country: us
Re: Applications of Sparse L1 Pursuits to Precision Reference
« Reply #46 on: April 17, 2018, 03:11:27 am »
In light of  @RandallMcRee's efforts, I think I should note that what I'm doing right now can just as well be done with least squares. The Marquardt-Levenberg implementation in gnuplot is particularly good. A solver has to be able to beat that for me to consider using it.  However, gnuplot is configured to do function approximations rather than as a general Ax=y solver, so it is more convenient to use the machinery I already have in place for L1 pursuits.  Because the thermal noise has Gaussian properties, L2  will fit it just as well as L1.

The sparse L1 pursuit doesn't offer a significant  advantage until you need to select one of a large number of choices.  That presents an underdetermined problem which L2 cannot handle.  If the residue after removal of the thermal noise shows significant deviation from a simple 1/f relationship, then a sparse pursuit becomes important.  But I'll need a bunch of data to resolve that.

At the moment I am just building and testing the machinery.

The takeaway from this post is L1 is *really* good at some things L2 can't do.  But for a *lot* of things L2 is just as good and several orders of magnitude cheaper to compute.
 

Offline rhb

  • Super Contributor
  • ***
  • Posts: 1603
  • Country: us
So, what do the plots show?
« Reply #47 on: April 18, 2018, 01:36:07 am »

How many angels can dance on the head of a pin?

The thermal noise amplitude spectrum shows the standard deviation of the thermal noise as a function of frequency after accounting for aliasing.  The other two plots indicate how high a frequency bleeds through the ADC and some notion of the numerical error of the modeling.  Actually quantifying the numerical error requires finding a solution to the series expansion.

However, you can't do anything about it.  It's just a precise statement of the uncertainty.  The same applies to the 1/f spectrum.  You can't correct for it except by collecting a large number of measurements and invoking the central limit theorem.

The sole benefit I can see at the moment is that having rigorous statements of the expected variance (i.e. the power spectrum) makes it possible to identify the presence of other noise sources in the measurement. Whether that is of significant practical benefit remains to be seen.

The ultimate goal of this project is to be able to accurately predict the aging curve for a voltage reference from data collected during the first 1000 or so hours of operation.  Properly quantifying the uncertainty created by the irreducible noise is a small, but essential step in that direction.
 

Offline zhtoor

  • Frequent Contributor
  • **
  • Banned!
  • Posts: 337
  • Country: pk
Re: Applications of Sparse L1 Pursuits to Precision Reference
« Reply #48 on: May 12, 2018, 10:46:14 pm »
Hello @rhb,

find attached a sample dataset of 10,000 samples, sampling frequency is 3.89Hz, data starts at line #10.
a scatter plot is also attached.

questions:
1. how well (and if?) can the next 1000 samples be predicted from the previous 1000?
2. this scatter plot does look sparse to me, how can we further "sparsify" it?
3. how would you specify the L1 pursuit problem for this dataset?

ref:
further details available at: http://www.eevblog.com/forum/metrology/vlfn-characterization-of-reference-devices-topologies/msg1534340/#msg1534340

best regards.

-zia
« Last Edit: May 12, 2018, 10:48:05 pm by zhtoor »
 

Offline rhb

  • Super Contributor
  • ***
  • Posts: 1603
  • Country: us
Re: Applications of Sparse L1 Pursuits to Precision Reference
« Reply #49 on: May 14, 2018, 06:05:48 am »
The series is not long enough.  An eyeball inspection indicates there is no significant trend which can be predicted.

There might be a small time dependent trend, but the quantization obscures it.

The first order test to apply would be to examine the autocorrelation aka power spectrum and compare that to a spike convolved with the window function imposed by the series lenth.
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf