EEVblog Electronics Community Forum

General => General Technical Chat => Topic started by: gildasd on May 18, 2016, 09:31:44 am

Title: Datalogging: correcting readout from a sensor that drifts
Post by: gildasd on May 18, 2016, 09:31:44 am
Hello, this is for my end of studies project:
I'm datalogging the force changes on a large cube due to wind. It does about 65000 read outs a day...
One of the sensors is drifting up and down with the voltage changes (Cube 3 column)
I'm trying to correct this using Excel, but I'm not sure in what direction to go...
I've tried "((Vcc-5.31+1)*Cube3)" and it works at the extremes but over correct the 5.03V values.
Not really sure what kind of methodology I should be using to correct this

datetime   temp   wind   cube2   cube3   vcc
 "2016/5/12 6:22:1"   18.32   0   332   342   5.31
 "2016/5/12 6:22:2"   18.32   0   333   340   5.31
 "2016/5/12 6:22:3"   18.32   0   332   342   5.31
 "2016/5/12 6:22:4"   18.32   0   314   337   5.31
 "2016/5/12 6:22:5"   18.32   0   332   341   5.31
 "2016/5/12 6:22:6"   18.32   0   331   343   5.31
 "2016/5/12 6:22:7"   18.32   0   335   344   5.31
 "2016/5/12 6:22:8"   18.32   0   322   340   5.31
 "2016/5/12 6:22:9"   18.32   1   341   344   5.31
 "2016/5/12 6:22:10"   18.32   3   337   341   5.31
 "2016/5/12 1:14:10"   22.19   0   316   281   5.03
 "2016/5/12 1:14:11"   22.51   0   317   283   5.03
 "2016/5/12 1:14:12"   22.51   0   313   286   5.01
 "2016/5/12 1:14:13"   22.19   0   348   289   5.03
 "2016/5/12 1:14:14"   22.19   0   345   288   5.03
 "2016/5/12 1:14:15"   22.19   0   305   279   5.03
 "2016/5/12 1:14:16"   22.19   0   331   282   5.01
 "2016/5/12 1:14:17"   22.19   4   355   288   5.03
 "2016/5/12 1:14:18"   22.19   3   319   278   5.03
 "2016/5/12 1:14:19"   22.19   1   321   282   5.03
 "2016/5/11 17:46:19"   35.08   0   314   162   4.25
 "2016/5/11 17:46:20"   35.08   0   331   166   4.25
 "2016/5/11 17:46:21"   35.08   0   333   170   4.25
 "2016/5/11 17:46:22"   35.08   0   304   159   4.25
 "2016/5/11 17:46:23"   35.08   0   323   167   4.25
 "2016/5/11 17:46:24"   35.4   0   335   170   4.25
 "2016/5/11 17:46:25"   35.4   1   334   166   4.25
 "2016/5/11 17:46:26"   35.08   2   311   161   4.27
 "2016/5/11 17:46:27"   35.08   1   315   164   4.25
 "2016/5/11 17:46:28"   35.4   1   340   171   4.25
Title: Re: Datalogging: correcting readout from a sensor that drifts
Post by: rs20 on May 18, 2016, 09:43:32 am
Why only 30 points of data?

Even with only 30 points, plotting cube3 vs vcc clearly shows that the relationship is non-linear. Take all your data, plot cube3 vs vcc, and fit a polynomial (or whatever shape makes sense). The equation of the fitted polynomial can then be used as your correction term.

See: https://docs.google.com/spreadsheets/d/1Fblyl6y5soteb-Dns6m3vamXS10_gbyAXF2jRYn7AOo/edit#gid=556521029
Title: Re: Datalogging: correcting readout from a sensor that drifts
Post by: gildasd on May 18, 2016, 04:17:07 pm
Thanks! I'll look into that.
And you are right, I'll an extra data point between the min and the average and  the max and the average.
For now I'm using ax+b variations.

The picture is where I'm at now, the min and mas are perfect, but I over-correct the medium values.
(https://www.eevblog.com/forum/chat/datalogging-correcting-readout-from-a-sensor-that-drifts/?action=dlattach;attach=225946;image)
Title: Re: Datalogging: correcting readout from a sensor that drifts (SOLVED)
Post by: gildasd on May 18, 2016, 08:24:22 pm
I think I found it.
The previous reply put me on the right track
It's well within the margin of error. Wonderful!

Cube3*(sqrt((5/Vcc)^6.5))

More a lucky guess than anything... Happy dance.
Title: Re: Datalogging: correcting readout from a sensor that drifts
Post by: rs20 on May 18, 2016, 10:13:33 pm
Seems like you'd be far better off doing a regression than just guessing formulae, but whatever works for you I guess...
Title: Re: Datalogging: correcting readout from a sensor that drifts
Post by: gildasd on May 19, 2016, 09:47:17 am
Quote from: rs20 on May 18, 2016, 10:13:33 pm
Seems like you'd be far better off doing a regression than just guessing formulae, but whatever works for you I guess...
Well, guessing with a bit of maths intuition. I wanted to stay in the y=ax+b complexity.
I started with what I know: the difference in Vcc.
Taken as an difference this would correct the base and the end of the curve.
This was somewhere but not perfect. A simple subtraction did not work.

So I decided to try all the fundamentals: addition, subtraction, multiplication, division, square and root.
Just by deduction on what the factor could be (in it's simplest form): addition, subtraction, multiplication, division, square and root.
The division of the "ideal voltage" of 5V by the detected Vcc gave me a factor that looked great, but how to use it?
Your formula was hinting at something like "ax^(bx-c)" that can be read as "(ax^bx)/(ax^c)"
This kinda reminded me of some square root results, so I returned to the division above with your idea of a control exponent: Sqrt((5V/Vcc)^x)
Frankly that looked too simple to work.
I was quite surprised that it did. The mantra "when it doubt, go back to fundamentals" does work!

Do you have a source for "real world" regressions?
Title: Re: Datalogging: correcting readout from a sensor that drifts
Post by: rs20 on May 19, 2016, 09:59:47 am
Quote from: gildasd on May 19, 2016, 09:47:17 am
Do you have a source for "real world" regressions?

Look at the sheet I linked. Excel, OpenOffice, even Google Sheets will fit a simple quadratic on a series of points and label the curve with the formula.
Title: Re: Datalogging: correcting readout from a sensor that drifts
Post by: gildasd on May 19, 2016, 11:09:02 am
Quote from: rs20 on May 19, 2016, 09:59:47 am
Quote from: gildasd on May 19, 2016, 09:47:17 am
Do you have a source for "real world" regressions?

Look at the sheet I linked. Excel, OpenOffice, even Google Sheets will fit a simple quadratic on a series of points and label the curve with the formula.
Ah, ok, I think it was taught with a totally different name in French, under what menu/help section of Excel should I start with (i am no expert)?
Title: Re: Datalogging: correcting readout from a sensor that drifts
Post by: rs20 on May 19, 2016, 11:20:13 am
Quote from: gildasd on May 19, 2016, 11:09:02 am
Ah, ok, I think it was taught with a totally different name in French, under what menu/help section of Excel should I start with (i am no expert)?

Plot the data as an X/Y chart/graph. Then, somewhere in the settings for the graph, you will be able to add a "regression" or "trend line" or something like that.

https://support.office.com/en-us/article/Choosing-the-best-trendline-for-your-data-1bb3c9e7-0280-45b5-9ab0-d0c93161daa8