Datalogging with Arduino

[Ian.M]

It depends on the level of power integrity you require.  However its not safe to assume that power will get cut cleanly in an outage without damaging PSUs or tripping breakers that will require manual resetting so even if the rest of the building power comes back on in an hour your logger could still be without power till you next inspect it.

If short duration backup is adequate, 6x fresh alkaline AA brand-named cells + two Schottky diodes to prevent the PSU charging them or the cells backfeeding the PSU, is good for uninterrupted power for >10 hours @100mA to 7V, with the requirement that the PSU voltage is greater than 9V during normal operation.

[gildasd]

It depends on the level of power integrity you require.  However its not safe to assume that power will get cut cleanly in an outage without damaging PSUs or tripping breakers that will require manual resetting so even if the rest of the building power comes back on in an hour your logger could still be without power till you next inspect it.

If short duration backup is adequate, 6x fresh alkaline AA brand-named cells + two Schottky diodes to prevent the PSU charging them or the cells backfeeding the PSU, is good for uninterrupted power for >10 hours @100mA to 7V, with the requirement that the PSU voltage is greater than 9V during normal operation.

We have a dedicated power unit just for our building. Because we have VERY expensive navigation and motor simulators, I can assure you that a shut down is going to be clean.
That being said, our uni got sternly told off a few years back, when a broken wire in ship sized electric motor in the workshop made it's power draw "infinity"... Melting said power unit in a few seconds.

[Ian.M]

How long was the power out for after the motor mishap?

It looks like the minimum cost non-rechargeable backup supply I described would probably be good enough.   If the workshop have any more meltdowns accounting for a gap in your data wont be a big problem . . .

[gildasd]

How long was the power out for after the motor mishap?

It looks like the minimum cost non-rechargeable backup supply I described would probably be good enough.   If the workshop have any more meltdowns accounting for a gap in your data wont be a big problem . . .

Less than one day; The harbour is a 24/7/365 operation... So they sent a generator in a container straight away (and some angry electricians).
But the workshop is now isolated should this happen again.

[Ian.M]

In that case, the minimum cost 6xAA backup will be fine.  Log the DC supply voltage and if its had more than 2 hours on batteries simply replace them.  Over here that would cost me £1 for 6x Kodak AA cells from the pound shop, and its not expected to happen often so any more expense is hard to justify.

[gildasd]

Has anyone a good recommendation introduction to Arduino programming?
The .CC class is not that good; a bit messy and missing steps even in their most basic stuff. Confusing and time wasting.
Just basic blocs... I'm already blinking multiple LEDs, but have no info beyond timing...

Website, Youtube channel etc... The support does not really matter.

Edit:
Doing this: https://opensourcehardwaregroup.com/thearduinocourse/

[gildasd]

...
Beware also of the precision of the RTC.
The one in the Adafruit shield  is the DS1307. This  is not very precise, with possible drift of a few mn per month.
 Depending on the  time accuracy you need,  you will need to use the DS3231 , or even
some GPS timing for ultimate precision, with no drift.
...

I think I've found a way to make the time drift irrelevant:
If the data from one Arduino is OK, I'm happy.

[Ian.M]

It depends how paranoid you are.  If a serious electrical fault on one Arduino could take out the OPAMP, it could affect the signal to the other Arduino.   One way round that is to use high linearity analog optocouplers (with appropriate signal conditioning circuits round them) for the analog inputs to both Arduinos.  Then a regulator failure on one cant affect the rest of the system.  However analog optoisolation isn't easy or cheap.

Another approach would be to apply an external synchronisation pulse to all the Arduinos.  That's a digital pulse so easy to isolate by optocoupling, and as long as the interval between sync pulses is greater than twice the maximum possible clock drift, as long as you log the sync pulse's timestamp on all Arduinos, you can reconstruct the data set in accurate temporal order.  If you want absolute timestamping, you'll need an external reference for the sync pulse (e.g. divide down a GPS 1pps signal), but for relative timestamping any long period periodic signal will do, even from a CMOS 555!  Alternatively, divide down the mains supply frequency, as in most developed countries, the grid operator controls the line frequency pretty closely and additionally attempts to balance the errors to keep the total number of cycles per day constant to support electromechanical clocks and timers that use a synchronous motor.  See: http://www.mainsfrequency.com/gridtime.htm

[gildasd]

It depends how paranoid you are.  If a serious electrical fault on one Arduino could take out the OPAMP, it could affect the signal to the other Arduino.   One way round that is to use high linearity analog optocouplers (with appropriate signal conditioning circuits round them) for the analog inputs to both Arduinos.  Then a regulator failure on one cant affect the rest of the system.  However analog optoisolation isn't easy or cheap.

Another approach would be to apply an external synchronisation pulse to all the Arduinos.  That's a digital pulse so easy to isolate by optocoupling, and as long as the interval between sync pulses is greater than twice the maximum possible clock drift, as long as you log the sync pulse's timestamp on all Arduinos, you can reconstruct the data set in accurate temporal order.  If you want absolute timestamping, you'll need an external reference for the sync pulse (e.g. divide down a GPS 1pps signal), but for relative timestamping any long period periodic signal will do, even from a CMOS 555!  Alternatively, divide down the mains supply frequency, as in most developed countries, the grid operator controls the line frequency pretty closely and additionally attempts to balance the errors to keep the total number of cycles per day constant to support electromechanical clocks and timers that use a synchronous motor.  See: http://www.mainsfrequency.com/gridtime.htm

I'll try to think of a small input protection circuit to accompany the opamp unity amplifier... If I get a spike, so what, it will get drowned in the sea of data...
Optocoupler or galvanic separation (for analogue) is pushing the protection to a place where I rather replace burnt out parts than deal with the implied complexity.

As for time stamping, I don't care if the two Arduinos are off by six hours after 3 months:
- Sensor A and B get sampled at the same timeX by Arduino A. Repeat every second. Store every minute.
- Sensor A and B get sampled at the same timeY by Arduino B. Repeat every second. Store every minute.
- I could add, for an extra 100€, another two or three redundant Arduinos for extra safety.
I then have two (or more) valid data sets, I only need one to accomplish my thesis.

[Ian.M]

Its your data, your thesis, and your money, and  as you noted, you could simply throw money at the problem and duplicate the whole logger enough times that failure is extremely unlikely, but its worth noting that as long as you've got enough storage capacity and processing speed, if the experiment is not convenient to repeat, it is better to log in more detail, including parameters that might be useful but you initially don't expect to need, rather than finding out three months down the line that you need data you haven't logged.   Without a sync signal, you cant reconstruct the data set easily/reliably in a multiple fault situation where you've lost different signals from different Arduinos.

It puts me in mind of the astronavigation student I once sailed with who was too lazy to work out his sights immediately after taking them.   He had the time, astronomical body, temperature (for refraction corrections), observed altitude, index error etc. all carefully logged and was going on about his intentions to do the sight reduction by computer once back ashore.  One small problem: he hadn't logged the E.P, nor did he have the speed and course steered data that would allow him to reconstruct an E.P. day by day by D.R for the sight reduction program or even reduce a simple Sun - run - Sun set of sights, and I don't think he had many if any of multiple star fixes or the occasionally possible Sun Moon fixes that could be reduced directly by spherical trig.  After three months of work, finding out from the newbies he was trying to impress that it was all useless and having that confirmed by the skipper didn't help his disposition in the slightest.   

I'm not trying to imply you are like that student in any way, I'm just pointing out that if you cant go back for a 'do-over', its gotta be right the first time, and you've got to anticipate *ALL* your future needs. 

A mains derived sync signal is *CHEAP*, (especially if you are already using a linear PSU and can get at one end of the secondary before the bridge rectifier), and is remarkably effective. 

Another cheap & easy sync signal option would be a lid switch on the weatherproof box I assume you are going to use.  That would tag the time each day you open it to swap SD cards.

If you don't care about isolation, the same sync signal can be fed each board via an isolation resistor of a few K.  If one board shorts out, as long as there is enough drive from the sync signal source, all the others will still get the signal.

Finally, a GPS module (as suggested by JacquesBBB) can be had for under 20 Euro and would give you an absolute time reference.

[gildasd]

Its your data, your thesis, and your money, and  as you noted, you could simply throw money at the problem and duplicate the whole logger enough times that failure is extremely unlikely, but its worth noting that as long as you've got enough storage capacity and processing speed, if the experiment is not convenient to repeat, it is better to log in more detail, including parameters that might be useful but you initially don't expect to need, rather than finding out three months down the line that you need data you haven't logged.   Without a sync signal, you cant reconstruct the data set easily/reliably in a multiple fault situation where you've lost different signals from different Arduinos.

It puts me in mind of the astronavigation student I once sailed with who was too lazy to work out his sights immediately after taking them.   He had the time, astronomical body, temperature (for refraction corrections), observed altitude, index error etc. all carefully logged and was going on about his intentions to do the sight reduction by computer once back ashore.  One small problem: he hadn't logged the E.P, nor did he have the speed and course steered data that would allow him to reconstruct an E.P. day by day by D.R for the sight reduction program or even reduce a simple Sun - run - Sun set of sights, and I don't think he had many if any of multiple star fixes or the occasionally possible Sun Moon fixes that could be reduced directly by spherical trig.  After three months of work, finding out from the newbies he was trying to impress that it was all useless and having that confirmed by the skipper didn't help his disposition in the slightest.   

I'm not trying to imply you are like that student in any way, I'm just pointing out that if you cant go back for a 'do-over', its gotta be right the first time, and you've got to anticipate *ALL* your future needs. 

A mains derived sync signal is *CHEAP*, (especially if you are already using a linear PSU and can get at one end of the secondary before the bridge rectifier), and is remarkably effective. 

Another cheap & easy sync signal option would be a lid switch on the weatherproof box I assume you are going to use.  That would tag the time each day you open it to swap SD cards.

If you don't care about isolation, the same sync signal can be fed each board via an isolation resistor of a few K.  If one board shorts out, as long as there is enough drive from the sync signal source, all the others will still get the signal.

Finally, a GPS module (as suggested by JacquesBBB) can be had for under 20 Euro and would give you an absolute time reference.

You are of course right.
With my thesis "leader" (I don't know the right word in English), we have decided to store "irrelevant data" such as temp and humidity, just in case...
And as an aside, there is an intend to leave part of the jig semi-permanently on the roof to store long term data (wind speed, buffeting, temp, humidity, light etc) for use by someone else a year or two down the line.

With this in mind, it's still early days, I need to get a good basic outline of my project in late September, and my UNi being specialised in all thing maritime is pretty good at making stuff fail-safe (all the relevant teachers are at least engineers and most have reached 1st Mech officer level...).
That being said, I'm really scared of overcomplexifying/over-engineering, because that's a common mistake I do. So I will probably have a input protection on the opamps and on the board power supplies (most common failures) and not go further unless asked.
I have a lot of sensor calibration work in CFD, wind tunnel and then analysing the data... That should be 80% of the work.

For the sync, I'll think about it, but doesn't that add an extra failure point?

The sync signal via resistor also seems a good idea.
GPS module would be good, but probably on a dedicated Arduino feed feeding a signal to the others...

I need to get my head around stuff and what they imply to the system on a whole.

(Sorry if this is incoherent, my 3 year old girl is jumping on my back as I type this)

[Ian.M]

Assuming you don't go for the GPS option (and given what I understand of your requirements, I wouldn't bother), the sync signal is simply a digital pulse that adds a timestamp line to the log on ALL the Arduiduinos simultaneously.  The ONLY way it can decrease reliability is if a fault causes it to continuously toggle so that it is requesting timestamps faster than they can be logged, and the solution to that is to write the software to ignore the sync input for a fairly long interval after each transition e.g. a 1 minute lockout would  result in a maximum of one line in 60 being sync timestamps even if the pin was toggling at 50Hz.

On the protection side of things: ask yourself "What happens if the Arduino crashes and the input pin(s) I'm using to read the sensor become set as output?"
Often a series resistor of a few hundred ohms between the sensor circuit and the Arduino pin is all that is needed to prevent a crash causing damage.

[gildasd]

On the protection side of things: ask yourself "What happens if the Arduino crashes and the input pin(s) I'm using to read the sensor become set as output?"
Often a series resistor of a few hundred ohms between the sensor circuit and the Arduino pin is all that is needed to prevent a crash causing damage.

That's why I'm using loads of unity amplifiers... The signal can only go one way (with 5V) and the amplification does not "rob" the original signal of power (if I understand correctly).

[Ian.M]

Done right, a unity gain amplifier isolates the signal source from loading effects.  Unfortunately there are many ways of doing it wrong and some can actually disturb the signal source.

Consider what happens if the power to the OPAMP fails.  Will its input load the signal source excessively? 

Will a fault voltage from the board it is feeding  exceed its maximum permitted input voltage differential and either damage the inputs or cause the other input to draw excessive current?

If it fails, will it short the input signal to ground or supply?

Many of these issues can be mitigated by using a large enough input resistor and a matching one in the feedback loop at the cost of a little bit of bandwidth, and slightly poorer noise performance.  If it fails in any way, with the resistors, worst case the signal source is loaded by the input resistor instead of being shorted to ground or the supply, and after the failure the effect of that loading on the signal can be calculated and, if the time of failure can be determined or reasonably estimated the data from the other logger can be processed to compensate for it.  The resistors also reduce the risk of damage to the OPAMP by preventing excessive input currents if it is not powered or during transients.

[gildasd]

Done right, a unity gain amplifier isolates the signal source from loading effects.  Unfortunately there are many ways of doing it wrong and some can actually disturb the signal source.

Consider what happens if the power to the OPAMP fails.  Will its input load the signal source excessively? 

Will a fault voltage from the board it is feeding  exceed its maximum permitted input voltage differential and either damage the inputs or cause the other input to draw excessive current?

If it fails, will it short the input signal to ground or supply?

Many of these issues can be mitigated by using a large enough input resistor and a matching one in the feedback loop at the cost of a little bit of bandwidth, and slightly poorer noise performance.  If it fails in any way, with the resistors, worst case the signal source is loaded by the input resistor instead of being shorted to ground or the supply, and after the failure the effect of that loading on the signal can be calculated and, if the time of failure can be determined or reasonably estimated the data from the other logger can be processed to compensate for it.  The resistors also reduce the risk of damage to the OPAMP by preventing excessive input currents if it is not powered or during transients.

I fully agree. Opamps need resistors for protection, we had that bashed into our unwilling heads in class 

[Ian.M]

I apologise for assuming you were unaware of the need for input resistors, but your quick sketch schematic in post #31 did omit them.   Also, its not uncommon for a SSI chip that's been blown by an output overload to be damaged badly enough that its input signals are affected. 

[gildasd]

I apologise for assuming you were unaware of the need for input resistors, but your quick sketch schematic in post #31 did omit them.   Also, its not uncommon for a SSI chip that's been blown by an output overload to be damaged badly enough that its input signals are affected.

Don't apologise, 12 months ago I would not have put them and would have cried later...