Author Topic: Data logging  (Read 792 times)

0 Members and 1 Guest are viewing this topic.

Offline Simon

  • Global Moderator
  • *****
  • Posts: 15105
  • Country: gb
  • Did that just blow up? No? might work after all !!
    • Simon's Electronics
Data logging
« on: March 23, 2019, 10:49:54 am »
I have a project that requires data logging to happen. It was suggested that the customer would plug the device in to a computer and download the data but I suspect tha the easiuer option is to write to a text files on an SD card that will be removed.

Now I have had SD cards fail on me in the past. Presumably I could put the SD card into standby between writes to reduce it's usage? Are there any controllers out there that deal with the raw data writing and manage the card for me?
 

Offline aldi

  • Contributor
  • Posts: 20
  • Country: nl
Re: Data logging
« Reply #1 on: March 23, 2019, 12:00:06 pm »
What platform are you using, microcontroller or something like a Raspberry?
 

Offline Psi

  • Super Contributor
  • ***
  • Posts: 7623
  • Country: nz
Re: Data logging
« Reply #2 on: March 23, 2019, 12:06:19 pm »
If you use SD/uSD cards you will need to do a FAT filesystem and that is more to go wrong.
If the device is bumped and the sd card contacts glitch while writing you may corrupt the filesystem.
Corrupted fat filesystems sometimes go undetected for quite some time as they continue to function but are overwriting data they shouldn't be.

So i wouldnt use a SD card unless you are sure its what you need.

If you can use a mcu with internal EEPROM this is much easier. (or add external eeprom)
If you have to use a mcu that only has FLASH this is a bit more complex as you can only erase things in page blocks, but it's still very doable.

One thing you will want to plan for is how you are going to handle a MCU reset occurring in the middle of a data write.
However your system works it will need to track that a file or write was completed correctly and handle if it detects one was not.
Checksum, start/end op flags etc.
« Last Edit: March 23, 2019, 12:09:35 pm by Psi »
Greek letter 'Psi' (not Pounds per Square Inch)
 

Offline Simon

  • Global Moderator
  • *****
  • Posts: 15105
  • Country: gb
  • Did that just blow up? No? might work after all !!
    • Simon's Electronics
Re: Data logging
« Reply #3 on: March 23, 2019, 01:19:56 pm »
What platform are you using, microcontroller or something like a Raspberry?

I'm currently planning to use an atmega 0 series.
 

Offline Simon

  • Global Moderator
  • *****
  • Posts: 15105
  • Country: gb
  • Did that just blow up? No? might work after all !!
    • Simon's Electronics
Re: Data logging
« Reply #4 on: March 23, 2019, 02:43:33 pm »
The main problem I have is how do I retrieve the data in a useful format so I thought SD card ant text files. I could use an eeprom.
 

Offline hamster_nz

  • Super Contributor
  • ***
  • Posts: 2296
  • Country: nz
Re: Data logging
« Reply #5 on: March 24, 2019, 08:27:11 pm »
I did this for a ESP32-based GPS logger (for logging a weeks worth of data to show how much it wanders).

My design
- buffered in RAM until it had at least a full 8kB buffer of data.
- Lit an LED for 5 seconds before the write
- Initialized the SD card stack and mounted the FAT32 filesystem
- Appended the data to the file.
- Closed and shut down the SD card stack
- Turned off the LED

It had a few buffers, so that if the SD was not present it could hold onto the data for a while until it run out of buffer space and an overflow occurs.

It also means you can swap to a new SD card any time you like (just as long as the warning light is off!)

You can find the source at https://github.com/hamsternz/esp32_serial_log_to_sd
« Last Edit: March 24, 2019, 08:29:17 pm by hamster_nz »
Gaze not into the abyss, lest you become recognized as an abyss domain expert, and they expect you keep gazing into the damn thing.
 

Offline Psi

  • Super Contributor
  • ***
  • Posts: 7623
  • Country: nz
Re: Data logging
« Reply #6 on: March 27, 2019, 11:29:37 pm »
Might be better to make a new file, copy old file to it with the new appended data. Then delete the old file and rename new file to same name.
That way a failure at any point is less likely to lose data.
Greek letter 'Psi' (not Pounds per Square Inch)
 

Offline thm_w

  • Super Contributor
  • ***
  • Posts: 2209
  • Country: ca
Re: Data logging
« Reply #7 on: March 28, 2019, 12:09:23 am »
Append data to the end of the text/csv file, and it should generally be OK if you lose power. What I did was also create a new file every day, and start writing to that, but you could do it more often if losing a days worth of data is too much (I don't recall ever losing anything). There should be a "flush" command you can send to the SD every 1min-10min (whatever is suitable for your application) to make sure data is written out and doesn't sit in cache too long (hamster_nz gives a good extreme example of this).
Having the card powered on/standby doesn't really cause wear on the card, neither does necessarily hammering the thing with writes (if they cache), but space them out just in case.

Spend a bit more on an endurance branded SD, making sure its genuine. Since 8GB would likely be plenty, it won't be very expensive.
If you want total overkill, can also use a 18650 based UPS module (~$3), which would keep power if mains supply is lost.

SD makes it very simple to recover the data from the device, if you don't have time or don't want to setup other interfaces (wifi/usb) for reading right now. IMO its the simple way to go.
 

Offline Simon

  • Global Moderator
  • *****
  • Posts: 15105
  • Country: gb
  • Did that just blow up? No? might work after all !!
    • Simon's Electronics
Re: Data logging
« Reply #8 on: March 28, 2019, 07:46:20 am »
I think an SD card controller will work off an SPI bus as would the EEPROM I am thinking so maybe I will make a plugin logging module that I can work on later as this is not an immediate requirement.
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf