A stored value beeing 4 bytes that make for a very big buffer !
But is that ok from a signal filtering point of view ? Excel datas from the sensor shows it working but is my idea a red flag.
package arduino;
/*
* Copyright (c) 2003, the JUNG Project and the Regents of the University
* of California
* All rights reserved.
*
* This software is open-source under the BSD license.
* See http://jung.sourceforge.net/license.txt for a description.
*/
public class DataMoments {
/**
* A data structure representing the central moments of a distribution including: <ul>
* <li> the mean </li>
* <li> the variance </li>
* <li> the skewness</li>
* <li> the kurtosis </li></ul> <br>
* Data values are each passed into this data structure via the accumulate(...) method
* and the corresponding central moments are updated on each call
*
* @author Didier H. Besset (modified by Scott White and, subsequently, by Leland Wilkinson)
*/
private double[] moments;
public DataMoments() {
moments = new double[5];
}
public void accumulate(double x) {
if (Double.isNaN(x) || Double.isInfinite(x))
return;
double n = moments[0];
double n1 = n + 1;
double n2 = n * n;
double delta = (moments[1] - x) / n1;
double d2 = delta * delta;
double d3 = delta * d2;
double r1 = n / n1;
moments[4] += 4 * delta * moments[3] + 6 * d2 * moments[2] + (1 + n * n2) * d2 * d2;
moments[4] *= r1;
moments[3] += 3 * delta * moments[2] + (1 - n2) * d3;
moments[3] *= r1;
moments[2] += (1 + n) * d2;
moments[2] *= r1;
moments[1] -= delta;
moments[0] = n1;
}
public double mean() {
return moments[1];
}
public double count() {
return moments[0];
}
public double kurtosis() {
if (moments[0] < 4)
return Double.NaN;
double kFact = (moments[0] - 2) * (moments[0] - 3);
double n1 = moments[0] - 1;
double v = variance();
return (moments[4] * moments[0] * moments[0] * (moments[0] + 1) / (v * v * n1) - n1 * n1 * 3) / kFact;
}
public double skewness() {
if (moments[0] < 3)
return Double.NaN;
double v = variance();
return moments[3] * moments[0] * moments[0] / (Math.sqrt(v) * v * (moments[0] - 1) * (moments[0] - 2));
}
public double standardDeviation() {
return Math.sqrt(variance());
}
public double variance() {
if (moments[0] < 2)
return Double.NaN;
return moments[2] * moments[0] / (moments[0] - 1);
}
}
But much much less resource hungry!
But it has smooth cut-off slope on frequency response and pretty mediocre rejection ratio at stop bandand, which is not suitable for many DSP applications.
But it has smooth cut-off slope on frequency response and pretty mediocre rejection ratio at stop bandand, which is not suitable for many DSP applications.
Yes, the selectivity and stopband attenuation are not good.
However, you won't find a FIR filter of the same length that has a better (lower) equivalent noise bandwidth (ENBW).
Do client required 400Hz update rate? If not, you can make sliding window not by sampling rate, but update rate. Just accumulate SamplingRate/UpdateRate samples in one before putting it into buffer.
What does he want to do with these numbers? There's no point of just storing the numbers in memory, they must be consumed somehow. How?
How much RAM is available?
If there's enough then use it, and show them how bad it behaves (abrupt change will take 30 seconds to register fully).
I think what is more useful is adaptive filtering, where after an abrupt change a simple low-pass filter is used, but once the signal is stable a wide moving average can then take over to reduce noise. Something like that.
Perhaps its best to figure out what the requirements actually are for performance.
If your MCU supports SPI, you could always use e.g. Microchip 23LC512 (524288 bits = 65536 bytes).
For the moving average, you only read one word and write one word per ADC sample, i.e. 400 reads (56 SPI clock cycles each) and 400 writes (56 SPI clock cycles each) per second: no need for DMA or hardware external memory support at all. Essentially, you'd use the SRAM as a cyclic buffer of 16384 words. By controlling how many words are between the write/head and the read/tail addresses, you can choose any window size between 2 and 16384 (16383).
You do need a 32+14 = 46-bit accumulator, too, with each new ADC sample added to it (and each oldest sample subtracted from it when the desired window length has been reached). The running average is then the 46-bit accumulator sum divided by the number of samples summed. At 400 samples/second, the maximum window duration would be 40.96 seconds. Note that you do need a division operation, 46-bit unsigned integer divided by a 16-bit unsigned integer, yielding a 32-bit unsigned integer (with overflow impossible), but this should not be a problem in practice, as most compilers provide 64-bit support (uint64_t) which works perfectly well for this.
> Actually no the update rate for the communication would be ~200ms.
So, 30 sec * 5 data-pack/sec = 150 records. Size of sliding window buffer should be 150 records.
> If I understand correctly you would average all sample between 2 communication send
Yes.
> What would be the downside of this ?
No downside. Result will be absolutely the same, as with buffer for all samples (30*400 = 12000)
You required to send only 1/80 of all sampled points, so there is no reason to store all 80 samples - you never send out 79 of them. Sum them right at sampling time.
#define SETS 150
uint64_t sum_total;
uint32_t sum_count;
uint32_t average;
volatile uint32_t set_sum[SETS];
volatile uint8_t set_overflow[SETS];
volatile uint8_t set_count[SETS];
uint8_t set_index;
uint8_t set_size;
This takes 18+6×SETS = 918 bytes of RAM. Initially, all these are cleared to all zeroes, except set_size to SETS; it sets the window duration in number of updates (0.2 second units) between 2 and SETS, inclusive. It is okay to initialize everything again to all zeros (except set_size to the new window duration), so that previous measurements are completely ignored and a new averaging window is started from scratch. if (__builtin_add_overflow(set_sum[set_index], SAMPLE, &(set_sum[set_index]))
set_overflow[set_index]++;
set_count[set_index]++;
or equivalent, i.e. add the sample value to the current sum, and increment overflow if the 32-bit value overflowed; and finally increment the current count. Normally, each set_count[] value will be between 0 and 80 (=400/5).) uint32_t state = begin_atomic();
uint8_t new_count = set_count[set_index];
uint32_t new_sum = set_sum[set_index];
uint8_t new_overflow = set_overflow[set_index];
if (++set_index >= set_size)
set_index = 0;
uint8_t old_count = set_count[set_index];
uint32_t old_sum = set_sum[set_index];
uint8_t old_overflow = set_overflow[set_index];
set_count[set_index] = 0;
set_sum[set_index] = 0;
set_overflow[set_index] = 0;
end_atomic(state);
sum_count += new_count;
sum_count -= old_count;
sum_total += new_sum + (uint64_t)(new_overflow) << 32;
sum_total -= old_sum + (uint64_t)(old_overflow) << 32;
if (sum_count > 0)
average = (sum_total + sum_count/2) / sum_count;
else
average = 0; // Do not report an average
where the +sum_count/2 adds rounding halfway upwards, instead of truncation towards zero. It is somewhat important that this occurs at regular 0.2 second intervals.What does he want to do with these numbers? There's no point of just storing the numbers in memory, they must be consumed somehow. How?The moving average filter requires that all values are stored ? Am I wrong about this ?
The moving average filter requires that all values are stored ?
As it's not quite unlikely to be a bit of an X-Y problem, maybe the OP can elaborate a bit on what the customer really wants to achieve - and if the OP doesn't fully know, I think they should ask.
From just the requirements - 400 Hz sampling, and a "long-term" moving average - my guess: it may be part of a machine that will measure low-frequency vibrations/oscillations around a baseline (baseline which would be determined by the moving average). Which is something that could possibly be tackled with a slightly different approach in terms of DSP.
Either that, or are they just willing to use a highish sample rate as oversampling, and a relatively large averaging window, in order to just increase the SNR?
The goal of the product is for the loadcell to report how much weight was lost is the field (imagine a fertilizer spreader) to know how much product was spread each seconds.
We want to know the "real" weight loss and not the motor, road etc noises.