Author Topic: Curve fitting question  (Read 6390 times)

0 Members and 1 Guest are viewing this topic.

Offline JesterTopic starter

  • Frequent Contributor
  • **
  • Posts: 898
  • Country: ca
Curve fitting question
« on: March 06, 2019, 12:34:03 pm »
I'm working on the calibration aspect of a project and would like to correct for some non linearity by applying a correction factor.

The uncorrected data and graph can be seen here (correction only at zero and full-scale):


I tried plugging the data into an online 3rd order Polynomial Regression tool and it helps but is far from ideal. We covered curve fitting in school decades ago, I don't recall much at this stage, except that a polynomial fit is likely a poor choice, perhaps Cubic spline or some other fit method?

Also can you suggest an online tool that will accept preferably 10-15 data pairs.

Thanks
« Last Edit: March 06, 2019, 12:37:01 pm by Jester »
 

Online magic

  • Super Contributor
  • ***
  • Posts: 7453
  • Country: pl
Re: Curve fitting question
« Reply #1 on: March 06, 2019, 12:41:56 pm »
online tool
:--

Octave/Matlab have polyfit, splinefit, etc.
Polynomial may suffice and has the advantage of being trivial to implement in embedded firmware, but you will need a much higher degree than 3 for all that curvy waviness.
 

Offline Siwastaja

  • Super Contributor
  • ***
  • Posts: 9327
  • Country: fi
Re: Curve fitting question
« Reply #2 on: March 06, 2019, 01:37:48 pm »
A strange curve. It may be difficult to fit a function. Of course you can, but I would consider using a lookup table instead.

Curve fitting works great when your error function is some easy, quick-to-calculate function with only one or two parameters - first order linear with offset and gain being the most typical example, and a second-order parabolic function often being the most complexity you want to deal with, especially in embedded. This tends to be the case when there is one dominant physical mechanism of error, over all other causes.

Lookup tables are most flexible, if you can afford the space for storage. Piecewise linear interpolation is simple to implement if you can't fit the LUT in full resolution. The best thing is, the lookup table adapts automatically even if a particular unit behaves very differently for any reason. With curves, once you have selected which function to use, you are limited to what you can do by changing its input parameters (which you typically try to minimize) - this is harder to analyze.
 
The following users thanked this post: Jester

Online golden_labels

  • Super Contributor
  • ***
  • Posts: 1472
  • Country: pl
Re: Curve fitting question
« Reply #3 on: March 06, 2019, 01:54:47 pm »
You are calculating a relative error for 0, which is already an indication of a problem. 0 can have only three relative error values, none useful for calculations: 0, -∞ and +∞.

Using absolute error provides datapoints that reveal a nice curve with two separate,  tiny subgroups on the sides. Therefore you can use a piecewise approach. First few values, having no error, have correction of +0. Then there is the huge blue part that fits logarithmic curve with R²≈0.93. Then there is the problem with the final values (red): there is not enough of them to actually calculate any curve. The two options, the 2nd and 3rd order polynomials, give some reasonable correction, but I feel like they are going to fail as soon as you collect more data in that region. The blue part gets a better fit (R²≥0.96, not shown on the chart) if the values are moved to the logarithmic scale and higher-order polynomial (≥4) is used, but I am also expecting this to be overfitting for that particular data set.

But all the above make sense for that single, particular sample. If you need a general approach for your system, those fixed values will not work. While the middle, blue part seems like it may work with logarithmic curve for any sample, the green and the red ones will probably not fit the models I have chosen for them if you collect samples from a different specimen or under different conditions.

Also, if you are implementing this in a microcontroller with limited resources, consider approximating the logarithmic curve with straight sections. Calculating logarithms is expensive.
« Last Edit: March 06, 2019, 01:57:56 pm by golden_labels »
People imagine AI as T1000. What we got so far is glorified T9.
 
The following users thanked this post: rs20, Jester

Online SiliconWizard

  • Super Contributor
  • ***
  • Posts: 15795
  • Country: fr
Re: Curve fitting question
« Reply #4 on: March 06, 2019, 03:13:04 pm »
I suggest considering Lagrange polynomials: https://en.wikipedia.org/wiki/Lagrange_polynomial
 
The following users thanked this post: Jester

Offline Henrik_V

  • Regular Contributor
  • *
  • Posts: 160
  • Country: de
  • “ground” is a convenient fantasy
Re: Curve fitting question
« Reply #5 on: March 06, 2019, 03:20:12 pm »
If you don't have a model of that error (resulting in a function with parameters to fit)
the (in my eyes) only valid error compesation is a lookup table with linear interpolation inbetween.
Greetings from Germany
Henrik

The number you have dialed is imaginary, please turn your phone 90° and dial again!
 
The following users thanked this post: rs20, Jester

Online golden_labels

  • Super Contributor
  • ***
  • Posts: 1472
  • Country: pl
Re: Curve fitting question
« Reply #6 on: March 06, 2019, 04:08:55 pm »
I suggest considering Lagrange polynomials: https://en.wikipedia.org/wiki/Lagrange_polynomial
Jester has only a small number of points and we also can’t tell if the variation is due to random effects or actual, fixed properties of the system. There is simply not enough data to make any conclusions about why values jump up and down; just look at all those waves interpolated at the original chart: there is nothing between inflection points and local extrema. And there is only one sample. The independent variable is not even spaced evenly¹, so all the hills and valleys may be merely effects of quite arbitrary choices.

In that situation it makes no sense to use any detailed function — there is no data to support it. However, using Lagrange polynomials would provide exactly that: a very detailed function built on top of data that isn’t there. It would just cause overfiting. Perhaps statistical approach is like building castles on sand, but using anything that provides a perfect fit is like building castles on air.

And if going for a perfect fit, I would rather use what Henrik_V suggested: LUT with linear interpolation. Much cheaper implementation, probably equally good/bad.

--- edit
Lagrange polynomials may work to some degree, if points are chosen wisely. But we do not know the actual model, so we can’t select those points. The data provided uses arbitrary locations.
____
¹ On either linear or logarithmic scale.
« Last Edit: March 06, 2019, 04:20:59 pm by golden_labels »
People imagine AI as T1000. What we got so far is glorified T9.
 
The following users thanked this post: Jester

Offline Conrad Hoffman

  • Super Contributor
  • ***
  • Posts: 2077
  • Country: us
    • The Messy Basement
Re: Curve fitting question
« Reply #7 on: March 06, 2019, 06:04:50 pm »
Wealth of stuff here you might try- http://zunzun.com/
 
The following users thanked this post: pardo-bsso

Offline aldi

  • Contributor
  • Posts: 28
  • Country: nl
Re: Curve fitting question
« Reply #8 on: March 06, 2019, 06:07:51 pm »
The "function finder" on zunzun.com may be useful.
 
The following users thanked this post: Jester

Offline den

  • Contributor
  • Posts: 41
  • Country: lv
  • Automotive engineer
Re: Curve fitting question
« Reply #9 on: March 06, 2019, 08:06:55 pm »
Calibration is performed by measuring the same thing with another equipment, which is pore accurate, but still has an error. The curve should include these error bars.
Anyway, I would never try to fit such a curve in this way, like with a cubic spline (which, by the way, you see in the graph - Excel uses it for smooth line option), because it just doesn't make sense. Just measure same thing on a different day and you will get a different curve.
 

Offline Kasper

  • Frequent Contributor
  • **
  • Posts: 791
  • Country: ca
Re: Curve fitting question
« Reply #10 on: March 07, 2019, 07:08:34 am »
I would get more data before curve fitting. If possible / feasible, test at different temperautres, battery levels, or whatever might affect your errors.  Also do some tests with Vin increasing and some with it decreasing, if possible.

It's hard to imagine an error pattern looking like that every time. If it varies on other tests, look for a pattern, if you see one, make an equation or a look up table. If there is too much difference between each test, then clean up your test set or your unit under test.

I'm sure there are better solutions but I've found excel works well for curve fitting. Sometimes I use the polynomials excel recommends and other times I make an equation and graph it then tune the equation until the graph matches the curve I'm trying to fit.
 
The following users thanked this post: Jester

Offline jbb

  • Super Contributor
  • ***
  • Posts: 1265
  • Country: nz
Re: Curve fitting question
« Reply #11 on: March 07, 2019, 08:19:30 am »
If it’s embedded, +1 for lookup table and piecewise linear interpolation.

However, that curve is weird. Might I suggest you examine your equipment to check for unexpected conditions?

Some colleagues of mine once had a hell of a time with a wonky rotary encoder. It produced very very strange output. Eventually they realised that the grub screw holding the encoder disk onto the shaft wasn’t done up.
 
The following users thanked this post: tggzzz, Jester

Offline voltsandjolts

  • Supporter
  • ****
  • Posts: 2547
  • Country: gb
Re: Curve fitting question
« Reply #12 on: March 07, 2019, 08:29:21 am »
Data looks strange, if it's not repeatable then it's not even worth trying to curve fit to it.
 

Offline rs20

  • Super Contributor
  • ***
  • Posts: 2322
  • Country: au
Re: Curve fitting question
« Reply #13 on: March 07, 2019, 08:34:08 am »
A Lagrange polynomial is a HORRIBLE idea in this context because the resulting polynomial will take a circuitous path to travel through every point precisely, no matter how ridiculous the resulting values between the given points are:



The ONLY time to use Lagrange polynomials is if you absolutely must have a polynomial curve that goes precisely through all your points, and absolutely literally couldn't care less what the rest of the values are.

In general I'd say linear interpolation is the best you're going to manage, except:

Data looks strange, if it's not repeatable then it's not even worth trying to curve fit to it.

+1 to this. The recorded data wildly changes direction from one sample to the next. If the wild changes are due to noise, you shouldn't try to fit to the noise. If the wild changes are a real, actual signal, then you need to make more finely spaced measurements to establish the true shape of the curve (and avoid Aliasing).
« Last Edit: March 07, 2019, 08:56:18 am by rs20 »
 
The following users thanked this post: Jester

Offline tggzzz

  • Super Contributor
  • ***
  • Posts: 21225
  • Country: gb
  • Numbers, not adjectives
    • Having fun doing more, with less
Re: Curve fitting question
« Reply #14 on: March 07, 2019, 08:40:43 am »
I'm working on the calibration aspect of a project and would like to correct for some non linearity by applying a correction factor.

The uncorrected data and graph can be seen here (correction only at zero and full-scale):


I tried plugging the data into an online 3rd order Polynomial Regression tool and it helps but is far from ideal. We covered curve fitting in school decades ago, I don't recall much at this stage, except that a polynomial fit is likely a poor choice, perhaps Cubic spline or some other fit method?

Also can you suggest an online tool that will accept preferably 10-15 data pairs.

Thanks

The "right" curve to use can only be defined after you know the theory of how the measurement is expected to vary. In the absence of that understanding, plotting a curve is like a Rorschach test :)

Examples:
  • if you are plotting mortality against time, then you ought to have a bathtub curve :)
  • if you are plotting noise, then you shouldn't use a curve, but should use mean and standard deviation :)
  • if you are plotting a time series where there is an expected temperature dependence, then you should use the measured temperature as the curve
  • if you are plotting a warm-up transient, perhaps an offset exponential curve is relevant
There are lies, damned lies, statistics - and ADC/DAC specs.
Glider pilot's aphorism: "there is no substitute for span". Retort: "There is a substitute: skill+imagination. But you can buy span".
Having fun doing more, with less
 
The following users thanked this post: rs20, Jester

Offline Henrik_V

  • Regular Contributor
  • *
  • Posts: 160
  • Country: de
  • “ground” is a convenient fantasy
Re: Curve fitting question
« Reply #15 on: March 07, 2019, 09:10:20 am »
Overseen the relative part... so +1 for golden_labels

fitting: https://xkcd.com/2048/

For a meaningful correction much more information is needed.
Are different ranges involved? (Should have individual LUTs)
Repeatability?
Stability/Drift?
Systematic errors (loaded voltage divider?)?
Greetings from Germany
Henrik

The number you have dialed is imaginary, please turn your phone 90° and dial again!
 
The following users thanked this post: Jester

Offline iMo

  • Super Contributor
  • ***
  • Posts: 5570
  • Country: va
Re: Curve fitting question
« Reply #16 on: March 07, 2019, 11:23:26 am »
The uncorrected data and graph can be seen here (correction only at zero and full-scale):
What are the Vdisp values at Vin=65 or 76 or 44? You do not know.
It could be anything.
You have to live with an error and you may fit with a polynomial such the resulting error will still be acceptable.
Readers discretion is advised..
 
The following users thanked this post: Jester

Offline mjkuwp

  • Supporter
  • ****
  • Posts: 260
  • Country: us
  • mechanical engineering defector
    • The Mz Lab
Re: Curve fitting question
« Reply #17 on: March 07, 2019, 12:15:20 pm »
It will help if you can post actual data in a text file rather than an image.

I suspect that the odd 'shape' of the curve is not real it is noise in the instrument and you would not want to try to calibrate this out.  A 2nd or 3rd order curve may be enough.

Do you have multiple samples of this data?  Or, could you take 10 samples at each reference point, average these and use those as the inputs to the calibration.

For something like this I would use either Python with numpy and matplotlib or EXCEL or even Google Sheets ( part of Google Docs)

in Google Sheets you would use Charts to make a Scatter plot and then add a trendline to a series and select the option for the Label of that trendline to show the equation.
 
The following users thanked this post: Jester

Offline JesterTopic starter

  • Frequent Contributor
  • **
  • Posts: 898
  • Country: ca
Re: Curve fitting question
« Reply #18 on: March 07, 2019, 09:10:06 pm »
Wealth of stuff here you might try- http://zunzun.com/

Thanks for the link, this in an excellent tool.  I ended up breaking the data into a few ranges and then applying correction factor per range. Displayed error is now < +/- 0.1% from 0-250V, and that's good enough for this project. I still need to test over temperature range and will add an additional tweak factor to compensate if needed.  :) :-+
 

Offline JesterTopic starter

  • Frequent Contributor
  • **
  • Posts: 898
  • Country: ca
Re: Curve fitting question
« Reply #19 on: March 07, 2019, 09:15:16 pm »
A Lagrange polynomial is a HORRIBLE idea in this context because the resulting polynomial will take a circuitous path to travel through every point precisely, no matter how ridiculous the resulting values between the given points are:



The ONLY time to use Lagrange polynomials is if you absolutely must have a polynomial curve that goes precisely through all your points, and absolutely literally couldn't care less what the rest of the values are.


Good point, and I observed what you described when I tried that approach.
 

Offline T3sl4co1l

  • Super Contributor
  • ***
  • Posts: 22436
  • Country: us
  • Expert, Analog Electronics, PCB Layout, EMC
    • Seven Transistor Labs
Re: Curve fitting question
« Reply #20 on: March 08, 2019, 01:15:28 am »
You'll need a quite high polynomial to fit that lumpy mess.

Also, is it actually lumpy, or is it truly noisy?  Is this an ADC input transfer function, by any chance?  Typically many-bits code crossings exhibit spikes, for which you could use a curve to minimize peak error, but not really get the bulk of it without just using a lookup table on the whole damn thing.

A procedural function could produce blips around specific breakpoints, without costing too much execution time or data space, but may cost a fair amount of code itself.

Most embedded platforms don't have div or float (many don't have mul either!), which makes anything but plain polynomial functions awkward.

Rational functions can fit spiky data very well indeed for a given order, but require at least p+q multiplies and one division (for rationals of the form A / B, for polynomials A and B, of order p and q respectively), assuming you don't run into problems with numerical range and stability, or other optimization options, that may suggest another factorization (sums of terms p/q + r/s + ..).

Tim
Seven Transistor Labs, LLC
Electronic design, from concept to prototype.
Bringing a project to life?  Send me a message!
 
The following users thanked this post: Jester

Offline AlcidePiR2

  • Regular Contributor
  • *
  • Posts: 95
  • Country: fr
Re: Curve fitting question
« Reply #21 on: March 08, 2019, 05:17:52 am »
As with Lagrange interpolation,  large degree polynomial is a very bad idea.

The problem of this plot is that most probably a large component is noise. You dont want to interpolate precisely noise. On the contrary, you want to use the least degree possible,  to make the correction robust.

Without knowing better the setting, it is difficult to give precise advice, but  I would dump the two extreme part of the curve, and conserve only  the [10:28] data points, where there is an obvious trend.

The just fit a line by quadratic regression.
One get cor = -0.663145 + 0.001981 * V

The two attached plots are   (V, error) and (V, error-cor).

Then only you wonder if you want to add additionnal corrections for the  extreme parts
 
The following users thanked this post: Jester

Offline AlcidePiR2

  • Regular Contributor
  • *
  • Posts: 95
  • Country: fr
Re: Curve fitting question
« Reply #22 on: March 08, 2019, 05:46:55 am »
Looking more closely to the data, I see the problem.

The  plotted error is the relative error. This is bad, as for very  small values it is meaningless.
One should plot the absolute errors.

Then  it appears that for large values, there was probably a reading problem on the scale. I would then discard these values.
A quadratic fit is then very adequate (see the plot) .  The plot is with the absolute error.
The fit is  F = -    0.005786 * V + 1.199683E-5 * V^2


The last curve  is  ERR-F vs V.
I believe that the  5 last points are bad measures.



 
The following users thanked this post: Jester

Offline Berni

  • Super Contributor
  • ***
  • Posts: 5050
  • Country: si
Re: Curve fitting question
« Reply #23 on: March 08, 2019, 07:06:25 am »
I say don't bother with curve fitting and just use a lookup table with interpolation.

Curve fitting works great when your graph is a smooth curve all the way (for example NTK temperature sensor corrections) as such a curve can be made into a polynomial with only 2 to 4 elements. While storing a precise lookup table would require something like 100 points. Speed of lookup is about the same on both, but the polynomial uses less memory. However when your curve is so erratic it would take a awfully long polynomial to aproximate it to a point where the two basically look the same. This ruins the advantage of polynomials as now you need to store a long list of elements in memory anyway while calculating the polynomials becomes significantly slower than a lookup table (And as you found curve fitting becomes hard at this point).

The trick to having a lookup table work just as well is in interpolation. For a smooth and dense graph you can use simple linear interpolation between points that's fast to compute too. If the points are too far apart then a bit of extra math can be added to do cubic interpolation. This looks at the data points around the point of interest to make a smooth spline trough them. This is computationally a little more expensive (About 6 multiplies and 10 sums) but still very fast.

Here is a nice page on interpolation: http://paulbourke.net/miscellaneous/interpolation/
 
The following users thanked this post: Jester

Offline AlcidePiR2

  • Regular Contributor
  • *
  • Posts: 95
  • Country: fr
Re: Curve fitting question
« Reply #24 on: March 08, 2019, 07:56:19 am »
I say don't bother with curve fitting and just use a lookup table with interpolation.

Berni, I dont agree with you. You need first to distinguish what part is due to the moise. You dont want to interpolate or use look out table for noise. It is pointless.

In the given example. I bet the level of noise is about 0.1 %.
 
The following users thanked this post: Jester


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf