Data logging

[Simon]

I have a project that requires data logging to happen. It was suggested that the customer would plug the device in to a computer and download the data but I suspect tha the easiuer option is to write to a text files on an SD card that will be removed.

Now I have had SD cards fail on me in the past. Presumably I could put the SD card into standby between writes to reduce it's usage? Are there any controllers out there that deal with the raw data writing and manage the card for me?

[aldi]

What platform are you using, microcontroller or something like a Raspberry?

[Psi]

If you use SD/uSD cards you will need to do a FAT filesystem and that is more to go wrong.
If the device is bumped and the sd card contacts glitch while writing you may corrupt the filesystem.
Corrupted fat filesystems sometimes go undetected for quite some time as they continue to function but are overwriting data they shouldn't be.

So i wouldnt use a SD card unless you are sure its what you need.

If you can use a mcu with internal EEPROM this is much easier. (or add external eeprom)
If you have to use a mcu that only has FLASH this is a bit more complex as you can only erase things in page blocks, but it's still very doable.

One thing you will want to plan for is how you are going to handle a MCU reset occurring in the middle of a data write.
However your system works it will need to track that a file or write was completed correctly and handle if it detects one was not.
Checksum, start/end op flags etc.

[Simon]

What platform are you using, microcontroller or something like a Raspberry?

I'm currently planning to use an atmega 0 series.

[Simon]

The main problem I have is how do I retrieve the data in a useful format so I thought SD card ant text files. I could use an eeprom.

[hamster_nz]

I did this for a ESP32-based GPS logger (for logging a weeks worth of data to show how much it wanders).

My design
- buffered in RAM until it had at least a full 8kB buffer of data.
- Lit an LED for 5 seconds before the write
- Initialized the SD card stack and mounted the FAT32 filesystem
- Appended the data to the file.
- Closed and shut down the SD card stack
- Turned off the LED

It had a few buffers, so that if the SD was not present it could hold onto the data for a while until it run out of buffer space and an overflow occurs.

It also means you can swap to a new SD card any time you like (just as long as the warning light is off!)

You can find the source at https://github.com/hamsternz/esp32_serial_log_to_sd

[Psi]

Might be better to make a new file, copy old file to it with the new appended data. Then delete the old file and rename new file to same name.
That way a failure at any point is less likely to lose data.

[thm_w]

Append data to the end of the text/csv file, and it should generally be OK if you lose power. What I did was also create a new file every day, and start writing to that, but you could do it more often if losing a days worth of data is too much (I don't recall ever losing anything). There should be a "flush" command you can send to the SD every 1min-10min (whatever is suitable for your application) to make sure data is written out and doesn't sit in cache too long (hamster_nz gives a good extreme example of this).
Having the card powered on/standby doesn't really cause wear on the card, neither does necessarily hammering the thing with writes (if they cache), but space them out just in case.

Spend a bit more on an endurance branded SD, making sure its genuine. Since 8GB would likely be plenty, it won't be very expensive.
If you want total overkill, can also use a 18650 based UPS module (~$3), which would keep power if mains supply is lost.

SD makes it very simple to recover the data from the device, if you don't have time or don't want to setup other interfaces (wifi/usb) for reading right now. IMO its the simple way to go.

[Simon]

I think an SD card controller will work off an SPI bus as would the EEPROM I am thinking so maybe I will make a plugin logging module that I can work on later as this is not an immediate requirement.