datalogging sampling frequency bottleneck

[Skyland]

My datalogging system which consits of the Flora, RTC, SD card breakout board, LSM9DS0 sensor is shown in the picture below;

https://www.dropbox.com/s/d8pwcn59nq11lzs/2014-11-25%2012.26.56.jpg?dl=0

The sampling rate/s is capped at 30 when writing into the sd card. If I disable the sensor and only save the time it goes up to 65. Writing into the serial interface rather than the sd card makes the rate goes up even higher at 127. for my application I need at least 100 samples/s. I am thinking the sd card is the problem after making the tests mentioned. The flora clock at 16Mhz is surely able to provide a higher sampling frequency. After checking the sensor datasheet and library I haven't found any limitation either. any input on this issue?

[Opticalworm]

Hiya Skyland,

Without looking at your code or flow diagram of your code I can only assume that you are opening a file on the SD card (say Temp.dat) and writing directly into it and closing the file when you are done.

If so, are you keeping the file open between samples or are you opening and closing it between samples?

The second will add extra overhead to each sample you make which could explain the slower rate.

-Ron

[Jeroen3]

Looking into my glass orb or truth, this is most likely the cause:
SD cards, because of their flash, prefer to be written in blocks equal to (or a number of times larger) than their minimum erase size. Mostly 512 bytes or entire sectors of a few kB.
If you need to write 1 byte to something not 0xFF (erased status of flash), you'd actually need to erase an entire flash page before writing it again. And erasing takes a lot of time.

If your filesystem/disk driver does not have a buffer for this, you're going to have bad time.
Not to mention a lot of unnecessary writes to the FAT table (you're using fat right?) wearing out the card in turbo mode.

[PA0PBZ]

+1

Jeroen was just a bit ahead of me 

I'd do the sampling on an interrupt and store the samples in memory. Then in a seperate process check the size of the buffer and save to SD if you think the time has come to do so.
But even then I would be worried about the SD card's life...

[dannyf]

I need at least 100 samples/s

You will have to figure out if it is burst of reads or sustained, and how much each read takes up in memory.

The simplest solution would be to buffer the reads in ram and send / save it via interrupts / dma.

If the bottleneck is the adc, you could use faster adc or multiple adc and sample in parallel. Some chips allow ping-pong styled adc and that helps a lot as well.

If all failed, you could look to optimize your code.

[Jeroen3]

While searching about that little mcu board wearable you have there. I noticed it has a ATmega32U4 on it.

This has certain limitation.
- float types. Not going to happen.
- sprintf is slow.
- sd card is in spi emulation mode. (only 1 of the 4 data channels)
- There is no dma
- Arduino does not have the most optimized pieces of code.
Banish software delays from all code you are using. You have 4 timers, use them.