Datalogging with Arduino

[gildasd]

I'm trying to get my head around using an Arduino for my Bachelor's Thesis project (called Mémoire in Belgium).
I've NEVER done anything with an microcontroller, so I'm a tiny bit nervous on splurging my hard earned student cash...
I'm not looking for someone to do this for me, I'm looking for help at avoiding the NOOB pitfalls.

This summer, I've decided to start to learn basic programming for the Arduino and build a basic setup.

I need to datalog quite a few things, the method is not the objective, so I can use off the shelf stuff.
My ultimate target will be to log once every second 4 analogue sources (pressure gauges, temp, humidity) and one digital (wind speed) 24/7 for 3 months.
Open source is better, because I am meant to pay for this (can't use that old copy of Excel they gave me at work in 2003) and provide a detailed BOM.
For this test I'll just be logging 1 analogue and 1 digital source. keeping stupid simple.

I've been reading stuff on Arduino.cc and going over tutorials...
... and for the initiated, it's slightly confusing.

To run this test, this is my simplified BOM:
1) Arduino board
2) USB SD card shield.
3) My sensors.
4) Small OEM 5V power supply
5) Real time controller?
6) Open office Calc.

To be able to replace anything in a few hours, in case of a major magic smoke moment, I'll be sourcing locally (and I can't put "some dude in China on Ebay" in my BOM).
I have a shop here in Gent (home) and a couple in Antwerp (Antwerp), so I can get stuff in the time it takes me to get there Monday to Saturday.

1) For the board my choices are limited to; Leonardo, UNO rev3, NANO (328) and M0.
I quite like the Nano due to it's form factor and 8 analogue inputs (I can have redundant sensors).
But none of these have an SD card reader...
2) The only SD shield I can get easily here is  a Wireless SD shield...
I obviously won't be using the wireless: http://www.gotron.be/media/files/Downloads/A000065.pdf
Or would this Adafruid SD breakout board be compatible? http://www.gotron.be/en/bouwkits/adafruit/adashield/micro-sd-card-breakout-board.html
3) Sensors: for the test I'll just wire a photo-resistor (anaogue) and a 555 timer circuit with another photo-resistor (digital; quantity of 1 and 0's, per unit of time, go up and down during the day), just crap to generate a signal.
4) 5V, 1A, plug style mains power supply.
5) An RTC: do I need this to get the Arduino to save data at regular intervals?
6) Open Office CALC is a simple spreadsheet, I'll have to write a script to transform the raw data into usable figures.
When I fooled around with GPS data, I would do this with a TCX file. It seems it's not that complicated to get an Arduino to do something similar: https://www.arduino.cc/en/Tutorial/Datalogger
The trick seems to open and close file each time. I'll have to code a bit to get it to open a new file every day, so if there is random corruption, I'll only lose a day of data.
I'll be retrieving the data via the USB port once a day.

Another option would be ditch most of this and use an Adafruit Datalogger: http://www.gotron.be/en/bouwkits/adafruit/adashield/adafruit-assembled-data-logging-shield-for-arduino.html

Feel free to comment, even if you think I'm doing it all wrong and should use Raspberry Pi instead (my only other realistic option)...


 

[Thor-Arne]

I'd suggest:

1. Arduino UNO
2. SD-RTC shield
3. 7-12V PSU
4. Sensors

The Arduino has a great support community, UNO being among the most popular boards.
The PSU must be more since it have to be more than the dropout of the internal regulator.

[Ian.M]

The sole point of using something like an Arduino rather than a Raspberry PI or other full featured Linux SBC is to minimise space/weight  and power consumption.  The software development effort required will certainly be greater on the Arduino platform.

Assuming the backup power issues to ensure continuous logging steer you towards the Arduino (as providing longterm battery backup to a Pi is a real bitch due to its much higher current consumption and restricted supply voltage range), you'd be an idiot not to get the Adafruit datalogger shield.  RTC, full size SD card, and a significantly larger prototyping area + example code specifically for data logging from Adafruit, what's not to like?

You'll simplify your life by writing the data directly to a file in CSV format which any spreadsheet and most databases can open directly and swapping SD cards daily rather than messing around trying to get USB access to the SD card in the logger working.  Use a push button to request SD card removal (so your sketch can close files etc.) and add an indicator LED for 'Safe to remove card'.  You can probably buffer a minute's or half a minute's data in RAM (which is a good idea anyway to reduce the number of SD card writes  and reduce the risk of card corruption) so provided you swap cards promptly, you wouldn't even loose any data.   I would suggest rotating log files on an hourly basis, it reduces the risk of data loss and makes file sizes more managgeable for post-processing.

You can feed an Arduino board's regulator direct from a 12V lead acid battery and keep the battery charged with a cheap float charger or even a solar panel if operating outdoors at a remote site.   Even a small SLA battery could power your board and sensors for days, giving you plenty of time to sort out any power interruptions without loosing data.

[ez24]

6) Open office Calc.

long time ago this gave me problems (sorry I forgot what) so I now use

LibreOffice Calc

just an alternative

[gildasd]

Thanks for all the answers...

Just writing stuff down for public consumption helps crystallise ideas into something realistic.
By the end of typing, I was already thinking along these lines...  Your answers helped confirm my choices; so I think I'll go:
1. Arduino UNO
2. Adafruit data logging shield
3. 7-12V PSU (with a backup battery on the "real" project) - even if the whole thing will use only about 100mA...
4. Sensors

This seems to offer the shortest and safest development pipeline.

[SL4P]

As usual, there are helpful and not-so-helpful comments flying all over the place.
A 2GB SD card (typical default support for Arduino) holds a lot of sample data.
As mentioned,I f you sample your inputs and timestamp them to file, you'll get a decent chunk of logging.
The power requirements are something you just have to manage, but there are several approaches for a three month trial.

So let's look at the storage...
4 x analog (2 byte / 10 bit) samples = 8 bytes
1 x digital count value (long int) =  4 bytes
- and the (long int) timestamp = total of 16 bytes per sample period

So extrapolating that out to 2GB,
yields a sample rate of <very fast>, probably more than you need.

[Ian.M]

Lets do the maths properly on a CSV file.
Each 10 bit ADC reading needs  four decimal digits.  Each unsigned long int needs ten decimal digits.  Each value except the last needs a comma after it.  Finally you need a <CR><LF> end of line code.  So that is 5*4+11*2+1 which equals 43 bytes per record, and, logging once per second is 3600 records per hour, or 154800 bytes.
One 2GB SD card would be good for over 18 months worth of data, but it would still be a good idea to swap cards at least weekly to reduce the risk of data loss.

Obviously, the Arduino's MCU has limited RAM so you should convert to decimal ASCII when you write to the file rather than when the sensor readings are buffered in RAM.

[SL4P]

Good point, I was counting for binary data storage...
I'm guessing you don't trust SD cards!  😉

[ez24]

Adafruit data logging shield

good luck

discontinued at Adafruit and out of stock at Maker

[retrolefty]

Adafruit data logging shield

good luck

discontinued at Adafruit and out of stock at Maker

 Are you sure. Adafruit presently showing in stock?
http://www.adafruit.com/products/1141

[Ian.M]

Good point, I was counting for binary data storage...
I'm guessing you don't trust SD cards!  😉

Its not that I don't trust them, though its well known that like other FLASH block devices they don't do well with frequent small writes,  its that I prefer to make pessimistic design decisions as I know that I if I do, I am likely to be pleasantly surprised by the final real world performance/reliability.

[ez24]

 

I forgot to check the details of this

http://www.adafruit.com/products/243

which shows this

https://www.adafruit.com/product/1141

[Kappes Buur]

You could check out http://www.instructables.com/ for an Arduino based weatherstation
to get some idea on how to bring your idea to fruition.

[gildasd]

Lets do the maths properly on a CSV file.
Each 10 bit ADC reading needs  four decimal digits.  Each unsigned long int needs ten decimal digits.  Each value except the last needs a comma after it.  Finally you need a <CR><LF> end of line code.  So that is 5*4+11*2+1 which equals 43 bytes per record, and, logging once per second is 3600 records per hour, or 154800 bytes.
One 2GB SD card would be good for over 18 months worth of data, but it would still be a good idea to swap cards at least weekly to reduce the risk of data loss.

Obviously, the Arduino's MCU has limited RAM so you should convert to decimal ASCII when you write to the file rather than when the sensor readings are buffered in RAM.

Never thought of it that way
Even adding a header and footer every hour, it's still 3x more than I'll ever need. You saved me buying a 16Gb SD card!

[gildasd]

Good point, I was counting for binary data storage...
I'm guessing you don't trust SD cards!  😉

Its not that I don't trust them, though its well known that like other FLASH block devices they don't do well with frequent small writes,  its that I prefer to make pessimistic design decisions as I know that I if I do, I am likely to be pleasantly surprised by the final real world performance/reliability.

Depending on what Arduino I can get today (forgot about the mega block party called Gentse Feeste...), the idea is to write every minute, ten minutes or hour.
The internal mem is there for something !

[gildasd]

Adafruit data logging shield

good luck

discontinued at Adafruit and out of stock at Maker

 Are you sure. Adafruit presently showing in stock?
http://www.adafruit.com/products/1141

There are still a few of the older green version in stock, but I'll be getting the new blue version.

[gildasd]

You could check out http://www.instructables.com/ for an Arduino based weatherstation
to get some idea on how to bring your idea to fruition.

I've seen that, but I am resisting reading it until I breadboard something simple...

(Sorry about the multiple posts; I'm answering from my bed with my IPhone 4. My first full week working on industrial electric motors makes me fully apreciate an extra few minutes on my back...)

[Psi]

Data-logging for a long period adds extra complexity over a typical MCU projects.

ie, how do you test if something is going to work correctly for 3 months without actually running it for 3 months?

If getting the data is very important you could have 2 Arduinos logging the sensor data together. Start the 2nd one maybe 1 week after the first.
If any coding bugs causes the 1st arduino to stop working the 2nd will still be running for a week and hopefully you will be checking it regularly and can restart the 1st one. eg..

These are the sort of things you need to consider when data logging for months.

[gildasd]

Data-logging for a long period adds extra complexity over a typical MCU projects.

ie, how do you test if something is going to work correctly for 3 months without actually running it for 3 months?

If getting the data is very important you could have 2 Arduinos logging the sensor data together. Start the 2nd one maybe 1 week after the first.
If any coding bugs causes the 1st arduino to stop working the 2nd will still be running for a week and hopefully you will be checking it regularly and can restart the 1st one. eg..

These are the sort of things you need to consider when data logging for months.

That's why I intend to so a 3 month test now 

[Ian.M]

If you are as pessimistic as another student I collaborated with on a long term datalogger project, you will duplicate the sensors as well with independent connections.   Admittedly he was measuring ground and buried sample temperatures with Dallas 1-wire sensors, which significantly increased the risk of sensor failure and he needed months of uninterrupted data for his thesis.  We went lo-tech brute force for the power supply:  a couple of good condition 12V car batteries from a scrap yard + a 9V Alkaline 6xAA backup battery bank to allow the car battery to be swapped out for charging without loosing power to the logger.  Its worth logging main battery voltage so you can spot any developing problems before total failure occurs.

However, if the logger software is properly designed there will be very few things that can crash it short of hardware failure or external EMI.  The hourly log file rotation keeps file sizes manageable so you aren't likely to run into any filesystem bugs there.   It would be advisable to either rotate log file folders on a weekly basis so you don't run into problems with the number of files per folder, or quick format the card that is being swapped into the logger to wipe previous logs.

Software testing may need to include manipulating the RTC and SD card filesystem to artificially advance the system state to where it would be after days or months of continuous logging.

One hack to the Adafruit that would be worth looking at would be to lift the EN pin (3) of the 3.3V regulator for the SD card, and wire it to an Arduino I/O pin so SD card power is under software control.  Being able to cycle power to the SD card increases the chances of recovering from a 'hung' card if it fails on write.

Log any fatal errors to internal EEPROM so you can attempt a post-mortem analysis in more detail than "Its dead, Jim ...".

[gildasd]

If you are as pessimistic as another student I collaborated with on a long term datalogger project, you will duplicate the sensors as well with independent connections.   Admittedly he was measuring ground and buried sample temperatures with Dallas 1-wire sensors, which significantly increased the risk of sensor failure and he needed months of uninterrupted data for his thesis.  We went lo-tech brute force for the power supply:  a couple of good condition 12V car batteries from a scrap yard + a 9V Alkaline 6xAA backup battery bank to allow the car battery to be swapped out for charging without loosing power to the logger.  Its worth logging main battery voltage so you can spot any developing problems before total failure occurs.

However, if the logger software is properly designed there will be very few things that can crash it short of hardware failure or external EMI.  The hourly log file rotation keeps file sizes manageable so you aren't likely to run into any filesystem bugs there.   It would be advisable to either rotate log file folders on a weekly basis so you don't run into problems with the number of files per folder, or quick format the card that is being swapped into the logger to wipe previous logs.

Software testing may need to include manipulating the RTC and SD card filesystem to artificially advance the system state to where it would be after days or months of continuous logging.

One hack to the Adafruit that would be worth looking at would be to lift the EN pin (3) of the 3.3V regulator for the SD card, and wire it to an Arduino I/O pin so SD card power is under software control.  Being able to cycle power to the SD card increases the chances of recovering from a 'hung' card if it fails on write.

Log any fatal errors to internal EEPROM so you can attempt a post-mortem analysis in more detail than "Its dead, Jim ...".

Cheers mate!

I am pessimistic, I must have a 12 hour turnaround from a seagull eating it and then getting struck by lighting.

The project will reside on the roof of my uni and use mains. However, I'll probably add a small battery pack (few hours worth) because there is a major building site one lock and bridge away... So excrement will hit the ceiling mounted air moving device if I don't prepare for the brown mist.

I had not thought of the folder rotation, but that makes perfect sense: cameras (and other stuff) use this.

Just came back from the shop, the Arduino UNO box is tiny, you can't fit many matches in there.
Right now, I'll concentrate on getting "helloworld" noob tribulations done and dusted before moving back to datalogging in earnest.

[Ian.M]

The *MINIMUM* battery backup pack to consider for that sort of application is 100 hours duration.  If power fails shortly after your daily logger check and its the day before a weekend that has an adjacent public holiday, its quite likely that you may not discover the problem till the daily logger check on the next work day.

That's the attraction of a massive car battery - a couple of hundred ampere hours of reserve means you really don't have to worry much about sensor and MCU power management as long as you can keep the mean current down to a few tens of mA.  It also serves to stop your logger blowing off the roof during storms or being stolen by seagulls! (And your Belgian gulls - at least the ones around the Oostende fishmarket - are the largest and most agressive ones I have ever met!)

[JacquesBBB]

I  have been doing a similar long time  data logging project, but with the additional  difficulty that
I wanted everything to run on  a single LIPO battery for months.
In  this case,  5 V arduino is not good, and it is better to design  from scratch using 3.3 V  8 Mhz
Atmega328, in order to save all possible power, in the lines of what is done
by the jeelab http://jeelabs.org/.

But if you are on the main, your solution of using plain UNO should be OK.

For the SD cards, I put two of them,  with a switch between the two with leds that shows the one which is active.
This allows to remove the inactive one without any   loss of data.
Even if I take away the SD card for several hours.

Even if you are on the main, you need a backup power as small breakdown can occur. In my case, I suspect that this led
to some corruption of  the SD cards.

Beware also of the precision of the RTC.
The one in the Adafruit shield  is the DS1307. This  is not very precise, with possible drift of a few mn per month.
 Depending on the  time accuracy you need,  you will need to use the DS3231 , or even
some GPS timing for ultimate precision, with no drift.

But the best is to start with what you have, and let it evolve afterwards.
As it was already said, doubling  the logger is very useful, as you can improve one while continuing to
log data.
 

[Ian.M]

For meteorological purposes, sample timestamping sub-minute  absolute accuracy isn't likely to be important as long as the sampling rate is reasonably accurate.  If you log the timestamp when the 'request card removal' button is pressed, and carefully manually record the time you do so each day, you will be able to compensate for long-term clock drift when post-processing the data without increasing the hardware cost of your logger.   OTOH GPSes for RC plane/quadcopter appications are reasonably affordable, so if you need precision timing I'd add on one of those and correct the clock to it frequently.

[gildasd]

I'll be running two test jigs:

One "optimised" and one reference. As long as they drift the same, no big deal. Maybe I'll need to reset them both once a week?

As for battery life, my Uni is in Antwerp Harbour, that has a dedicated backup power-plant, so even if ISIS drives a semi full of fertiliser and bad intentions into the refinery's gates, we should get a blackout of minutes - in theory - 1 hour max in real life.

[Ian.M]

It depends on the level of power integrity you require.  However its not safe to assume that power will get cut cleanly in an outage without damaging PSUs or tripping breakers that will require manual resetting so even if the rest of the building power comes back on in an hour your logger could still be without power till you next inspect it.

If short duration backup is adequate, 6x fresh alkaline AA brand-named cells + two Schottky diodes to prevent the PSU charging them or the cells backfeeding the PSU, is good for uninterrupted power for >10 hours @100mA to 7V, with the requirement that the PSU voltage is greater than 9V during normal operation.

[gildasd]

It depends on the level of power integrity you require.  However its not safe to assume that power will get cut cleanly in an outage without damaging PSUs or tripping breakers that will require manual resetting so even if the rest of the building power comes back on in an hour your logger could still be without power till you next inspect it.

If short duration backup is adequate, 6x fresh alkaline AA brand-named cells + two Schottky diodes to prevent the PSU charging them or the cells backfeeding the PSU, is good for uninterrupted power for >10 hours @100mA to 7V, with the requirement that the PSU voltage is greater than 9V during normal operation.

We have a dedicated power unit just for our building. Because we have VERY expensive navigation and motor simulators, I can assure you that a shut down is going to be clean.
That being said, our uni got sternly told off a few years back, when a broken wire in ship sized electric motor in the workshop made it's power draw "infinity"... Melting said power unit in a few seconds.

[Ian.M]

How long was the power out for after the motor mishap?

It looks like the minimum cost non-rechargeable backup supply I described would probably be good enough.   If the workshop have any more meltdowns accounting for a gap in your data wont be a big problem . . .

[gildasd]

How long was the power out for after the motor mishap?

It looks like the minimum cost non-rechargeable backup supply I described would probably be good enough.   If the workshop have any more meltdowns accounting for a gap in your data wont be a big problem . . .

Less than one day; The harbour is a 24/7/365 operation... So they sent a generator in a container straight away (and some angry electricians).
But the workshop is now isolated should this happen again.

[Ian.M]

In that case, the minimum cost 6xAA backup will be fine.  Log the DC supply voltage and if its had more than 2 hours on batteries simply replace them.  Over here that would cost me £1 for 6x Kodak AA cells from the pound shop, and its not expected to happen often so any more expense is hard to justify.

[gildasd]

Has anyone a good recommendation introduction to Arduino programming?
The .CC class is not that good; a bit messy and missing steps even in their most basic stuff. Confusing and time wasting.
Just basic blocs... I'm already blinking multiple LEDs, but have no info beyond timing...

Website, Youtube channel etc... The support does not really matter.

Edit:
Doing this: https://opensourcehardwaregroup.com/thearduinocourse/

[gildasd]

...
Beware also of the precision of the RTC.
The one in the Adafruit shield  is the DS1307. This  is not very precise, with possible drift of a few mn per month.
 Depending on the  time accuracy you need,  you will need to use the DS3231 , or even
some GPS timing for ultimate precision, with no drift.
...

I think I've found a way to make the time drift irrelevant:
If the data from one Arduino is OK, I'm happy.

[Ian.M]

It depends how paranoid you are.  If a serious electrical fault on one Arduino could take out the OPAMP, it could affect the signal to the other Arduino.   One way round that is to use high linearity analog optocouplers (with appropriate signal conditioning circuits round them) for the analog inputs to both Arduinos.  Then a regulator failure on one cant affect the rest of the system.  However analog optoisolation isn't easy or cheap.

Another approach would be to apply an external synchronisation pulse to all the Arduinos.  That's a digital pulse so easy to isolate by optocoupling, and as long as the interval between sync pulses is greater than twice the maximum possible clock drift, as long as you log the sync pulse's timestamp on all Arduinos, you can reconstruct the data set in accurate temporal order.  If you want absolute timestamping, you'll need an external reference for the sync pulse (e.g. divide down a GPS 1pps signal), but for relative timestamping any long period periodic signal will do, even from a CMOS 555!  Alternatively, divide down the mains supply frequency, as in most developed countries, the grid operator controls the line frequency pretty closely and additionally attempts to balance the errors to keep the total number of cycles per day constant to support electromechanical clocks and timers that use a synchronous motor.  See: http://www.mainsfrequency.com/gridtime.htm

[gildasd]

It depends how paranoid you are.  If a serious electrical fault on one Arduino could take out the OPAMP, it could affect the signal to the other Arduino.   One way round that is to use high linearity analog optocouplers (with appropriate signal conditioning circuits round them) for the analog inputs to both Arduinos.  Then a regulator failure on one cant affect the rest of the system.  However analog optoisolation isn't easy or cheap.

Another approach would be to apply an external synchronisation pulse to all the Arduinos.  That's a digital pulse so easy to isolate by optocoupling, and as long as the interval between sync pulses is greater than twice the maximum possible clock drift, as long as you log the sync pulse's timestamp on all Arduinos, you can reconstruct the data set in accurate temporal order.  If you want absolute timestamping, you'll need an external reference for the sync pulse (e.g. divide down a GPS 1pps signal), but for relative timestamping any long period periodic signal will do, even from a CMOS 555!  Alternatively, divide down the mains supply frequency, as in most developed countries, the grid operator controls the line frequency pretty closely and additionally attempts to balance the errors to keep the total number of cycles per day constant to support electromechanical clocks and timers that use a synchronous motor.  See: http://www.mainsfrequency.com/gridtime.htm

I'll try to think of a small input protection circuit to accompany the opamp unity amplifier... If I get a spike, so what, it will get drowned in the sea of data...
Optocoupler or galvanic separation (for analogue) is pushing the protection to a place where I rather replace burnt out parts than deal with the implied complexity.

As for time stamping, I don't care if the two Arduinos are off by six hours after 3 months:
- Sensor A and B get sampled at the same timeX by Arduino A. Repeat every second. Store every minute.
- Sensor A and B get sampled at the same timeY by Arduino B. Repeat every second. Store every minute.
- I could add, for an extra 100€, another two or three redundant Arduinos for extra safety.
I then have two (or more) valid data sets, I only need one to accomplish my thesis.

[Ian.M]

Its your data, your thesis, and your money, and  as you noted, you could simply throw money at the problem and duplicate the whole logger enough times that failure is extremely unlikely, but its worth noting that as long as you've got enough storage capacity and processing speed, if the experiment is not convenient to repeat, it is better to log in more detail, including parameters that might be useful but you initially don't expect to need, rather than finding out three months down the line that you need data you haven't logged.   Without a sync signal, you cant reconstruct the data set easily/reliably in a multiple fault situation where you've lost different signals from different Arduinos.

It puts me in mind of the astronavigation student I once sailed with who was too lazy to work out his sights immediately after taking them.   He had the time, astronomical body, temperature (for refraction corrections), observed altitude, index error etc. all carefully logged and was going on about his intentions to do the sight reduction by computer once back ashore.  One small problem: he hadn't logged the E.P, nor did he have the speed and course steered data that would allow him to reconstruct an E.P. day by day by D.R for the sight reduction program or even reduce a simple Sun - run - Sun set of sights, and I don't think he had many if any of multiple star fixes or the occasionally possible Sun Moon fixes that could be reduced directly by spherical trig.  After three months of work, finding out from the newbies he was trying to impress that it was all useless and having that confirmed by the skipper didn't help his disposition in the slightest.   

I'm not trying to imply you are like that student in any way, I'm just pointing out that if you cant go back for a 'do-over', its gotta be right the first time, and you've got to anticipate *ALL* your future needs. 

A mains derived sync signal is *CHEAP*, (especially if you are already using a linear PSU and can get at one end of the secondary before the bridge rectifier), and is remarkably effective. 

Another cheap & easy sync signal option would be a lid switch on the weatherproof box I assume you are going to use.  That would tag the time each day you open it to swap SD cards.

If you don't care about isolation, the same sync signal can be fed each board via an isolation resistor of a few K.  If one board shorts out, as long as there is enough drive from the sync signal source, all the others will still get the signal.

Finally, a GPS module (as suggested by JacquesBBB) can be had for under 20 Euro and would give you an absolute time reference.

[gildasd]

Its your data, your thesis, and your money, and  as you noted, you could simply throw money at the problem and duplicate the whole logger enough times that failure is extremely unlikely, but its worth noting that as long as you've got enough storage capacity and processing speed, if the experiment is not convenient to repeat, it is better to log in more detail, including parameters that might be useful but you initially don't expect to need, rather than finding out three months down the line that you need data you haven't logged.   Without a sync signal, you cant reconstruct the data set easily/reliably in a multiple fault situation where you've lost different signals from different Arduinos.

It puts me in mind of the astronavigation student I once sailed with who was too lazy to work out his sights immediately after taking them.   He had the time, astronomical body, temperature (for refraction corrections), observed altitude, index error etc. all carefully logged and was going on about his intentions to do the sight reduction by computer once back ashore.  One small problem: he hadn't logged the E.P, nor did he have the speed and course steered data that would allow him to reconstruct an E.P. day by day by D.R for the sight reduction program or even reduce a simple Sun - run - Sun set of sights, and I don't think he had many if any of multiple star fixes or the occasionally possible Sun Moon fixes that could be reduced directly by spherical trig.  After three months of work, finding out from the newbies he was trying to impress that it was all useless and having that confirmed by the skipper didn't help his disposition in the slightest.   

I'm not trying to imply you are like that student in any way, I'm just pointing out that if you cant go back for a 'do-over', its gotta be right the first time, and you've got to anticipate *ALL* your future needs. 

A mains derived sync signal is *CHEAP*, (especially if you are already using a linear PSU and can get at one end of the secondary before the bridge rectifier), and is remarkably effective. 

Another cheap & easy sync signal option would be a lid switch on the weatherproof box I assume you are going to use.  That would tag the time each day you open it to swap SD cards.

If you don't care about isolation, the same sync signal can be fed each board via an isolation resistor of a few K.  If one board shorts out, as long as there is enough drive from the sync signal source, all the others will still get the signal.

Finally, a GPS module (as suggested by JacquesBBB) can be had for under 20 Euro and would give you an absolute time reference.

You are of course right.
With my thesis "leader" (I don't know the right word in English), we have decided to store "irrelevant data" such as temp and humidity, just in case...
And as an aside, there is an intend to leave part of the jig semi-permanently on the roof to store long term data (wind speed, buffeting, temp, humidity, light etc) for use by someone else a year or two down the line.

With this in mind, it's still early days, I need to get a good basic outline of my project in late September, and my UNi being specialised in all thing maritime is pretty good at making stuff fail-safe (all the relevant teachers are at least engineers and most have reached 1st Mech officer level...).
That being said, I'm really scared of overcomplexifying/over-engineering, because that's a common mistake I do. So I will probably have a input protection on the opamps and on the board power supplies (most common failures) and not go further unless asked.
I have a lot of sensor calibration work in CFD, wind tunnel and then analysing the data... That should be 80% of the work.

For the sync, I'll think about it, but doesn't that add an extra failure point?

The sync signal via resistor also seems a good idea.
GPS module would be good, but probably on a dedicated Arduino feed feeding a signal to the others...

I need to get my head around stuff and what they imply to the system on a whole.

(Sorry if this is incoherent, my 3 year old girl is jumping on my back as I type this)

[Ian.M]

Assuming you don't go for the GPS option (and given what I understand of your requirements, I wouldn't bother), the sync signal is simply a digital pulse that adds a timestamp line to the log on ALL the Arduiduinos simultaneously.  The ONLY way it can decrease reliability is if a fault causes it to continuously toggle so that it is requesting timestamps faster than they can be logged, and the solution to that is to write the software to ignore the sync input for a fairly long interval after each transition e.g. a 1 minute lockout would  result in a maximum of one line in 60 being sync timestamps even if the pin was toggling at 50Hz.

On the protection side of things: ask yourself "What happens if the Arduino crashes and the input pin(s) I'm using to read the sensor become set as output?"
Often a series resistor of a few hundred ohms between the sensor circuit and the Arduino pin is all that is needed to prevent a crash causing damage.

[gildasd]

On the protection side of things: ask yourself "What happens if the Arduino crashes and the input pin(s) I'm using to read the sensor become set as output?"
Often a series resistor of a few hundred ohms between the sensor circuit and the Arduino pin is all that is needed to prevent a crash causing damage.

That's why I'm using loads of unity amplifiers... The signal can only go one way (with 5V) and the amplification does not "rob" the original signal of power (if I understand correctly).

[Ian.M]

Done right, a unity gain amplifier isolates the signal source from loading effects.  Unfortunately there are many ways of doing it wrong and some can actually disturb the signal source.

Consider what happens if the power to the OPAMP fails.  Will its input load the signal source excessively? 

Will a fault voltage from the board it is feeding  exceed its maximum permitted input voltage differential and either damage the inputs or cause the other input to draw excessive current?

If it fails, will it short the input signal to ground or supply?

Many of these issues can be mitigated by using a large enough input resistor and a matching one in the feedback loop at the cost of a little bit of bandwidth, and slightly poorer noise performance.  If it fails in any way, with the resistors, worst case the signal source is loaded by the input resistor instead of being shorted to ground or the supply, and after the failure the effect of that loading on the signal can be calculated and, if the time of failure can be determined or reasonably estimated the data from the other logger can be processed to compensate for it.  The resistors also reduce the risk of damage to the OPAMP by preventing excessive input currents if it is not powered or during transients.

[gildasd]

Done right, a unity gain amplifier isolates the signal source from loading effects.  Unfortunately there are many ways of doing it wrong and some can actually disturb the signal source.

Consider what happens if the power to the OPAMP fails.  Will its input load the signal source excessively? 

Will a fault voltage from the board it is feeding  exceed its maximum permitted input voltage differential and either damage the inputs or cause the other input to draw excessive current?

If it fails, will it short the input signal to ground or supply?

Many of these issues can be mitigated by using a large enough input resistor and a matching one in the feedback loop at the cost of a little bit of bandwidth, and slightly poorer noise performance.  If it fails in any way, with the resistors, worst case the signal source is loaded by the input resistor instead of being shorted to ground or the supply, and after the failure the effect of that loading on the signal can be calculated and, if the time of failure can be determined or reasonably estimated the data from the other logger can be processed to compensate for it.  The resistors also reduce the risk of damage to the OPAMP by preventing excessive input currents if it is not powered or during transients.

I fully agree. Opamps need resistors for protection, we had that bashed into our unwilling heads in class 

[Ian.M]

I apologise for assuming you were unaware of the need for input resistors, but your quick sketch schematic in post #31 did omit them.   Also, its not uncommon for a SSI chip that's been blown by an output overload to be damaged badly enough that its input signals are affected. 

[gildasd]

I apologise for assuming you were unaware of the need for input resistors, but your quick sketch schematic in post #31 did omit them.   Also, its not uncommon for a SSI chip that's been blown by an output overload to be damaged badly enough that its input signals are affected.

Don't apologise, 12 months ago I would not have put them and would have cried later...