Long term (2 weeks+) data logging on microSD card

[AQUAMAN]

Hi all

I am running a setup on an Arm mbed microcontroller and it's for an experiment that will run for 2-3 weeks. There will be a set of values logged every 2 seconds.

I am logging data through the serial port in MATLAB, but I don't trust the computer to not randomly freeze or shut itself down for this entire time (and probably at the most critical moment). So I also have a backup by logging the data on an SD card.

What I am wondering is is whether its OK to open a file for writing on and SD card, and write continuously to it for a 2 week period? My program at the moment opens a new file for writing at the start, and then will keep it open the whole time, only to close it again when the experiment is over

Is there anything bad about doing this, like increasing the chances of file corruption or something? Or should I open and close the file every time I write to it? What if there is a power cut and everything turns off? I presume at this point all the data will be lost?

The reason I ask is that the line     file = fopen("/sd/data.txt", "w"); is taking over 0.1s to execute, and this makes it very difficult fitting it in the timing around the rest of my program

Any help appreciated

[thm_w]

Yes that should be no problem to write to it continuously for 2 weeks.
If your API allows it, there may be a way to "force" writes out to the card, so they aren't held in cache for too long, where they would be lost upon power off. I ended up writing every ~2s and forcing actual write every 30s or so. If power is lost then at most I saw that 30s of data gone.
I was using Fatfs: f_open, f_seek to end, then f_write, and f_sync to force writes.

You might consider creating a new file every day, with current date in the name. That may reduce the chance of corrupting one big data file, and limit loss to a single day (but I never saw this happen).
Also, make sure you are appending data to the end of the file, so if the program restarts it won't overwrite data.txt accidentally.

[Nick Novak]

Starting with a freshly formatted card may speed things up or at least ease the burden of scanning for free clusters when writing to the card.  You could also experiment with the time required to write out various sized chunks of data, you may find the time required is fairly constant up to a certain write size and it may make more sense to queue data and write it less frequently.

Closing the file or at least forcing it to sync periodically is good advice.

You should also keep in mind the 4G file size limit of FAT32.  I'm not sure how big your data entries are but that may be a consideration after 2 weeks.