help with rapid ADC data aquizition

[powerfly]

 I am trying to obtain ADC voltage readings at a rate of 10kHz with a resolution of 12 bits or more. This is to be done with a portable, small device, such as a microcontroller with a small power bank attached to it. I would like to be able to use this with 7 data channels, and if possible even up to 24 channels. I have been looking at microcontrollers to do the job, but I'm stuggling to find a suitable one (or maybe I just need guidance on how to use them properly).

7 channels of 12 bit data is 7*1.5 bytes = 10.5 bytes.
10kHz --> 105kb/s of data storage.
Ideally, at least 20s of data could be stored at this rate before a short interval and then repeat.

1. Are there any microcontrollers that would be able to write at these sorts of speeds straight to an SD card or memory stick? - Someone I know tried doing this with an Arduino uno and the data logging was far too slow. would a teensy 3.6 work as it has a native micro sd card slot?

2. If this does not work, is it possible to store data at this rate to flash memory on a micro controller? apparently it is possible with arduinos, although they do not have enough flash memory. A teensy 4.0 might just be able to do the job, although I am not sure how to write at this rate to the flash memory of a teensy, it has a different architechture to most arduinos which I think makes things more tricky. Are there any other micro controllers I should be looking at?

3. Is a microcontroller the best option? I have thought about possibly using a rasberry pi, although I have no experience with theese and basically no coding experience, which might mean it is tricky. The 4 Gb of ram is appealing though and if the right sampling rates are attainable this could be a great solution. Due to the size and probably more tricky coding this option seems less appealing than a micro controller atm.

Bear in mind I have very limited experience with coding, any help would be much appreciated!

[Wimberleytech]

Silicon Labs sells 8-bit micros with 12-bit ADCs on them.  The C8051F020 supports a sample rate up to 100ksps.  In addition, it has an on-board 8 input mux.

[JimRemington]

would a teensy 3.6 work

Yes, but there are only two independent ADCs. A Teensy 3.2 will work too.

[jhpadjustable]

1. Just about any 32-bit MCU with an onboard ADC could do it. For perspective, an STM32F103 (bluepill, etc.) has a 1Msps ADC with 8 or more inputs depending on package. If zero skew and zero error isn't a requirement, you can sample as many channels as you like and pass them along to your storage device, maybe do a bit of oversampling on the way. You can even use DMA and timers to handle all the grunt work for you. Teensy 3.x would probably be enough power for the job.

2. SD cards are pretty easy. They can be used in an SPI-like mode, in case your chosen MCU doesn't have a proper SDIO peripheral or you don't want to use it for some reason. Beware that flash lifetime is rated by the number of writes. After you exceed the limit, you may find data isn't being written correctly. Maybe that's not a big problem if you have 8GB of SD card that does its own write leveling. Speed might still be a concern. You would need to qualify your particular choice of SD card. If you really don't need non-volatile storage, you could use a pseudo-SRAM with a serial interface, for example.

3. SD cards are pretty simple. USB host is hard. The trouble with the Pi is that the multitasking OS gets in the way, and it's also a bit anemic as to peripherals.

[powerfly]

would a teensy 3.6 work

Yes, but there are only two independent ADCs. A Teensy 3.2 will work too.

Are you saying that a teensy 3.6 would fully be able to do the job, or that it has ADCs?

I think the ADCs on the teensy each have several channels so it could be used for more than 2 channels couldn't it?

1. Just about any 32-bit MCU with an onboard ADC could do it. For perspective, an STM32F103 (bluepill, etc.) has a 1Msps ADC with 8 or more inputs depending on package. If zero skew and zero error isn't a requirement, you can sample as many channels as you like and pass them along to your storage device, maybe do a bit of oversampling on the way. You can even use DMA and timers to handle all the grunt work for you. Teensy 3.x would probably be enough power for the job.

2. SD cards are pretty easy. They can be used in an SPI-like mode, in case your chosen MCU doesn't have a proper SDIO peripheral or you don't want to use it for some reason. Beware that flash lifetime is rated by the number of writes. After you exceed the limit, you may find data isn't being written correctly. Maybe that's not a big problem if you have 8GB of SD card that does its own write leveling. Speed might still be a concern. You would need to qualify your particular choice of SD card. If you really don't need non-volatile storage, you could use a pseudo-SRAM with a serial interface, for example.

3. SD cards are pretty simple. USB host is hard. The trouble with the Pi is that the multitasking OS gets in the way, and it's also a bit anemic as to peripherals.

Thank you for your in depth response, I think it is helping me figure things out

At the moment I'm going for a teensy 3.6 or 4.0, it sounds from your comment that you are reccomending a micro-controller over a rasberry Pi.

I'm not quite understanding what you are saying about micro-SD cards. I am happy to spend a bit of money on a decent micro-SD card if it will store the data at the rate I need, for this a teensy 3.6 might be ideal as it has a native micro-SD port. Are there any micro-SD cards that could be used in conjunction with a teensy (or other microcontroller) and get the desired data rate? If so, what kind of Micro SD card?

As for the volatile RAM thing - Is it simple to add on this 64mb of ram to a teensy, temporarily store data there and subsequently dump the memory to a card when a chunk of data has been collected? If so, it does not matter that it is volatile.

[radiolistener]

7 channels of 12 bit data is 7*1.5 bytes = 10.5 bytes.

May be MAX197 will fits your needs.
There is a module on aliexpress: https://www.aliexpress.com/item/32952117207.html
Didn't tried it, but looks good.

[jhpadjustable]

Pi-type SBCs don't generally have decent ADCs onboard. You'd have to use an external ADC or seven, and then deal with multiplexing among all the ports, initiating conversions on a tightly timed basis, among other things. A Pi just isn't going to be able to do all that very reliably while Linux is running, managing a filesystem and block device, etc. So the Teensy 3.6 is a good choice if the on-board ADCs are up to it.

Your data rates are on the order of compressed lossy video recording, which isn't a very demanding requirement at all. I doubt any but the absolute cheapest boot-sale cards would have trouble. On the other hand, the cards actually contain a small MCU that does the wear-leveling, serial protocol handling, and so forth, and that code is a bit of a wildcard. Testing is the only way to be sure. You can get some idea by copying a very large file to it through a USB card reader. As a guess, if you can get 300kB/sec into it (200% margin, or 100% margin if you spare yourself the effort of packing the extra 4 bits out of each word), you probably won't have any trouble in the final application. Do beware of OS caches giving you falsely favorable results when you test. Or just buy Sandisk and not worry

The RAM I pointed out is just an SPI device. You use it more or less the same as you would an SPI flash. But you've got more than enough memory on board the Teensy to buffer for a few seconds, maybe even enough to hot-swap the card during operation if you're quick about it, and it takes some careful software design to work with that external RAM.

[powerfly]

Pi-type SBCs don't generally have decent ADCs onboard. You'd have to use an external ADC or seven, and then deal with multiplexing among all the ports, initiating conversions on a tightly timed basis, among other things. A Pi just isn't going to be able to do all that very reliably while Linux is running, managing a filesystem and block device, etc. So the Teensy 3.6 is a good choice if the on-board ADCs are up to it.

Your data rates are on the order of compressed lossy video recording, which isn't a very demanding requirement at all. I doubt any but the absolute cheapest boot-sale cards would have trouble. On the other hand, the cards actually contain a small MCU that does the wear-leveling, serial protocol handling, and so forth, and that code is a bit of a wildcard. Testing is the only way to be sure. You can get some idea by copying a very large file to it through a USB card reader. As a guess, if you can get 300kB/sec into it (200% margin, or 100% margin if you spare yourself the effort of packing the extra 4 bits out of each word), you probably won't have any trouble in the final application. Do beware of OS caches giving you falsely favorable results when you test. Or just buy Sandisk and not worry

The RAM I pointed out is just an SPI device. You use it more or less the same as you would an SPI flash. But you've got more than enough memory on board the Teensy to buffer for a few seconds, maybe even enough to hot-swap the card during operation if you're quick about it, and it takes some careful software design to work with that external RAM.

Thanks, your answer has provided more insight yet again.

It sounds like the external RAM can be complicated and as I'm a begginer I'll try and avoid that.

I know a guy who tried reading to an SD card using an Arduino uno and a shield that had a memory card slot on it. He said that there was an issue because it's not simply transferring data so the write speed isn't the main limiting factor. He said that the uno has to open a file, store it on the card, close it, then write another file or something. because of this it could only store a reading (12 bits) every 2ms or so which is far too slow for the rate I'm trying to achieve. If this is the only way to do it, then the write speed isn't the rate limiting step and is almost irrelevant.

Is this just an issue with not coding correctly and there is a way just to keep a file open and steam all of your data into it for like 20s before closing that file and making a new one?

An alternative that I think you mentioned at the end of your comment is to store maybe like 0.5Mb of data on the native RAM and whilst the other code is still reading data in, to dump that 0.5 Mb to a memory card, is that what you were saying?

[ a guess a card like this is the best thing: https://www.amazon.co.uk/SanDisk-microSDHC-Memory-Adapter-Performance/dp/B073JWXGNT/ref=sr_1_4?adgrpid=52894980363&gclid=CjwKCAiAuK3vBRBOEiwA1IMhum67ArXvd2iUwrEJ6WjaHgV0Qy4URl70Pw1gM7EKH2hPYCLI53zoiBoCg7kQAvD_BwE&hvadid=259084611967&hvdev=c&hvlocphy=1006976&hvnetw=g&hvpos=1t1&hvqmt=e&hvrand=9654974697421066106&hvtargid=kwd-295891704216&hydadcr=28148_1752695&keywords=micro+sd+card&qid=1575712020&sr=8-4 it claims up to 98mb/s write speed]

[radiolistener]

you can do in on stm32 microcontroller. But the type of controller depends on your needs. If you're needs just to get date from adc and write it to sdcard, you can do it on a cheap two-dollar stm32 microcontroller. And you can use integrated adc.

[powerfly]

you can do in on stm32 microcontroller. But the type of controller depends on your needs. If you're needs just to get date from adc and write it to sdcard, you can do it on a cheap two-dollar stm32 microcontroller. And you can use integrated adc.

Can do what? I'm not exactly sure what you're replying to, if it's one of the questions in my post above, which one?

cheap integrated ADCs aren't usually 12 bit and are often not 10kHz either.

[Wimberleytech]

you can do in on stm32 microcontroller. But the type of controller depends on your needs. If you're needs just to get date from adc and write it to sdcard, you can do it on a cheap two-dollar stm32 microcontroller. And you can use integrated adc.

Can do what? I'm not exactly sure what you're replying to, if it's one of the questions in my post above, which one?

cheap integrated ADCs aren't usually 12 bit and are often not 10kHz either.

Restating my earlier post...the Silabs ADC IS 12 bits and 100ksps!

[jhpadjustable]

Your Arduino mate wasn't quite up to the challenge. He might have had better luck if he batched up a whole sector of data before writing it out, and writing binary instead of ASCII text. Still not bad for a single-threaded approach. I do maintain that you skip the bit-packing and just store your values in aligned 16-bit units, especially if you've got gigabytes of flash to play with, so that you'll never have to worry about alignment on the reading end or the writing end.

An alternative that I think you mentioned at the end of your comment is to store maybe like 0.5Mb of data on the native RAM and whilst the other code is still reading data in, to dump that 0.5 Mb to a memory card, is that what you were saying?

More or less, reserving some space to buffer some of the filesystem metadata. Take care that you don't surpass your filesystem's limitations. You might find it easier on the whole to work in terms round wall-clock time, maybe do 2-second chunks instead, which is more than enough time to create or append a file on the flash.

That card is great, no worries.

cheap integrated ADCs aren't usually 12 bit and are often not 10kHz either.

Welcome to the 32-bit MCU world.   Again, the hoary old STM32F103 has two 1Msps ADCs with a headline spec of 12 bits, which you could oversample by 8x (or 16x if you use both) in this application. Other, newer STM32 members are even better equipped, in case that's not enough margin for you.

[powerfly]

Your Arduino mate was doing it wrong. He would have had better luck if he batched up a whole sector of data before writing it out, and I bet he was also formatting the output as text instead of binary.  (But the Arduino fans don't usually think at this level...) I still recommend that you skip the bit-packing and just store your values in aligned 16-bit units, especially if you've got gigabytes of flash to play with, so that you'll never have to worry about alignment on the reading end or the writing end.

An alternative that I think you mentioned at the end of your comment is to store maybe like 0.5Mb of data on the native RAM and whilst the other code is still reading data in, to dump that 0.5 Mb to a memory card, is that what you were saying?

More or less, reserving some space to buffer some of the filesystem metadata. Take care that you don't surpass your filesystem's limitations. You might find it easier on the whole to work in terms round wall-clock time, maybe do 2-second chunks instead, which is more than enough time to create or append a file on the flash.

That card is great, no worries.

cheap integrated ADCs aren't usually 12 bit and are often not 10kHz either.

Welcome to the 32-bit MCU world.   Again, the hoary old STM32F103 has two 1Msps ADCs with a headline spec of 12 bits, which you could oversample by 8x (or 16x if you use both) in this application. Other, newer STM32 members are even better equipped, in case that's not enough margin for you.

Thanks, that sounds good. the thing with the arduino (and the STM32F103) is that you wouldn't be able to batch up lots of data before writing as their RAM is so small (2kB or 20kB). so whilst the old STM32F103 would be able to read the ADC at a high rate, how could it be storing the data at the high rate with 20s intervals? I mean, if I'm using 7 channels, 20kB would get used up in ~0.2 s. Am I misunderstanding something here?

I'm planning on using a Teensy 3.6 anyway that has 256K (I think that means 256KB) of RAM so that should be able to write then dump to the micro SD card via the native port without a problem (I think).

Also, if you could point me in the right direction regarding how to do all the coding, that would be super helpful as I'm a coding noob. I think I'd like to use the arduino IDE if possible as it seems simplest

[rstofer]

In theory, you can transfer multiple megabytes per second to an SD card depending on the model
https://www.expertreviews.co.uk/storage/1404380/how-to-choose-an-sd-card-class-and-speed-ratings-explained

But you probably need something faster than an Arduino to do it and the code probably needs to be tightened up A LOT!  DMA and multiple buffers will help.

Horsepower matters!  How about the Teensy 4?  It is blazing fast (600 MHz) has two 12-bit ADCs along with 14 analog pins.  This doesn't really matter because the ADCs could be external on an SPI or I2C bus and run through the DMA channel.  Given a bunch of internal memory, creating packets for the SD card will be very fast.

https://www.pjrc.com/store/teensy40.html

Not that I would necessarily go there but there is an Arduino compatible library for the chip called TeensyDuino which can be installed under the Arduino IDE although it escapes me why anyone would voluntarily use that IDE.  But the library installation includes the proper toolchain for the chip (ARM, not ATmega, obviously).  Keep the library, moved to anything else for the IDE.

The ARM chip itself:
https://www.nxp.com/docs/en/nxp/data-sheets/IMXRT1060CEC.pdf

[powerfly]

In theory, you can transfer multiple megabytes per second to an SD card depending on the model
https://www.expertreviews.co.uk/storage/1404380/how-to-choose-an-sd-card-class-and-speed-ratings-explained

But you probably need something faster than an Arduino to do it and the code probably needs to be tightened up A LOT!  DMA and multiple buffers will help.

Horsepower matters!  How about the Teensy 4?  It is blazing fast (600 MHz) has two 12-bit ADCs along with 14 analog pins.  This doesn't really matter because the ADCs could be external on an SPI or I2C bus and run through the DMA channel.  Given a bunch of internal memory, creating packets for the SD card will be very fast.

https://www.pjrc.com/store/teensy40.html

Not that I would necessarily go there but there is an Arduino compatible library for the chip called TeensyDuino which can be installed under the Arduino IDE although it escapes me why anyone would voluntarily use that IDE.  But the library installation includes the proper toolchain for the chip (ARM, not ATmega, obviously).  Keep the library, moved to anything else for the IDE.

The ARM chip itself:
https://www.nxp.com/docs/en/nxp/data-sheets/IMXRT1060CEC.pdf

I have been discussing using either the teensy 3.6 or 4.0 (see above comments). the advantage of the 3.6 is that it has the built in SD card reader and the processor is probably powerful enough. The 4.0 does have the added power and RAM, and I guess adding a card reader wouldn't be too difficult (although rapidly storing data to the card may be more difficult?)

When you say the code needs to tightened up a lot, how would one go about doing this?

jhpadjustable mentioned some stufff he thinks will help, although having someone point me in the right direction of how to do this would be helpful. I am very much a beginner when it comes to coding. the teensyduino library is appealing as it seems like it would be simple to code in.

[jhpadjustable]

Thanks, that sounds good. the thing with the arduino (and the STM32F103) is that you wouldn't be able to batch up lots of data before writing as their RAM is so small (2kB or 20kB). so whilst the old STM32F103 would be able to read the ADC at a high rate, how could it be storing the data at the high rate with 20s intervals? I mean, if I'm using 7 channels, 20kB would get used up in ~0.2 s. Am I misunderstanding something here?

You're missing nothing! You are absolutely correct. You would either have to write more often, like every 50-100ms or so still might be feasible, or choose another STM32 with more RAM (e.g. the STM32F3/STM32F4 range), or go with that external RAM for buffering with all that entails. I can't think of any STM32s that have worse specs on their ADCs than the F103, except that some in the STM32F0 range might come with only one unit instead of two. #justsayin on the chance you were making a bunch and wanted to push the cost down a bit.

I'm planning on using a Teensy 3.6 anyway that has 256K (I think that means 256KB) of RAM so that should be able to write then dump to the micro SD card via the native port without a problem (I think).

That sounds fine. You'll have plenty of room to work with if you drop the write interval to about 1/2 second. How long did you intend to leave this device in operation for, and how critical is it that you not lose data? I'm getting a little concerned about burning out the flash from overuse, which could happen if you write the metadata block back too often. If you intend to run for months, you might think about pre-allocating a big file on the flash, using the file data section as a big ring buffer, and not writing to the metadata at all.

This doesn't really matter because the ADCs could be external on an SPI or I2C bus and run through the DMA channel.

Why, they're free with the chip.

[radiolistener]

1. Are there any microcontrollers that would be able to write at these sorts of speeds straight to an SD card or memory stick?

yes, you can do it with STM32 microcontrollers. But they have different performance and different peripherals, so you're needs to choice right type of STM32 microcontroller.

3. Is a microcontroller the best option?

yes

[radiolistener]

cheap integrated ADCs aren't usually 12 bit and are often not 10kHz either.

cheap STM32 microcontrollers have 1-2 ADC with 12 bit resolution which can work with sample rate up to 1 MHz. And about 8 channels with analog multiplexer.

That's enough to capture all 8 channels with 10 kHz sample rate and write it to sdcard. Such solution will needs to share 1-2 ADC between 8 channels. For example you can use first ADC to digitize odd channels and second ADC to digitize even channels.

[rstofer]

In theory, you can transfer multiple megabytes per second to an SD card depending on the model
https://www.expertreviews.co.uk/storage/1404380/how-to-choose-an-sd-card-class-and-speed-ratings-explained

But you probably need something faster than an Arduino to do it and the code probably needs to be tightened up A LOT!  DMA and multiple buffers will help.

Horsepower matters!  How about the Teensy 4?  It is blazing fast (600 MHz) has two 12-bit ADCs along with 14 analog pins.  This doesn't really matter because the ADCs could be external on an SPI or I2C bus and run through the DMA channel.  Given a bunch of internal memory, creating packets for the SD card will be very fast.

https://www.pjrc.com/store/teensy40.html

Not that I would necessarily go there but there is an Arduino compatible library for the chip called TeensyDuino which can be installed under the Arduino IDE although it escapes me why anyone would voluntarily use that IDE.  But the library installation includes the proper toolchain for the chip (ARM, not ATmega, obviously).  Keep the library, moved to anything else for the IDE.

The ARM chip itself:
https://www.nxp.com/docs/en/nxp/data-sheets/IMXRT1060CEC.pdf

I have been discussing using either the teensy 3.6 or 4.0 (see above comments). the advantage of the 3.6 is that it has the built in SD card reader and the processor is probably powerful enough. The 4.0 does have the added power and RAM, and I guess adding a card reader wouldn't be too difficult (although rapidly storing data to the card may be more difficult?)

Have you looked at the Horsepower Chart - the 4.0 is more than 5 times as fast as the 3.6
https://www.pjrc.com/store/teensy40.html
Speed is good!

When you say the code needs to tightened up a lot, how would one go about doing this?

I haven't looked closely at the TeensyDuino libraries but the stuff for the ATmega chip doesn't use any features like DMA or fancy interrupt drivers.  Mostly because the chip doesn't have much.

You would want to be able to concurrently transfer values from DMA buffers to the SD card.  In a perfect world, the buffers would be exactly the size of a disk block (512 bytes) and every time the DMA filled one it would generate an interrupt to tell the SD write code to transfer more data to the card.

I don't know if the TeensyDuino library has most of this code.  It could because the 3.6 code might be quite similar but I wouldn't necessarily count on library code if I want raw speed over portability across platforms.  Bare metal programming comes to mind.

jhpadjustable mentioned some stufff he thinks will help, although having someone point me in the right direction of how to do this would be helpful. I am very much a beginner when it comes to coding. the teensyduino library is appealing as it seems like it would be simple to code in.

I do have a couple of Teensy 4.0 boards but I haven't played with them yet.  Some years back I used the RawHID code and a early Teensy to add switches, knobs and dials to Microsoft Flight Simulator using USB HID.  The code worked well so I have always been a fan of the Teensy products.

Sometimes, internal ADCs don't have the required performance and that's why I said it might not matter because you could use the 3 internal SPI buses (with 16 byte FIFOs) to connect to external ADCs with better performance.  I haven't looked to see that the SPI gadget connects through DMA but I'll bet it does.  So, basically, all of the ADC reading and transfer can be done in the background.  When a buffer gets filled, the code sets a flag to tell the mainline to  transfer the buffer to the SD card.  This too should be done by DMA so very little code execution is required.

There are many examples of using SD cards with one of the common FAT filesystems with the entire project running under FreeRTOS and while this is an advanced topic, it truly simplifies the code by cleaning up the inter-communications between reading ADCs and writing the SD.  Semaphores and queues (or buffer) plus some handshake signals control everything.

But, given an excess of horsepower, start simple and see how fast you can go.  If it meets your needs, there is no reason to dive deeper into the code.

However, having an SD adapter onboard makes the 3.6 very compelling.  I might start there and if I need more horsepower, I know where to get it.

[powerfly]

Thanks, that sounds good. the thing with the arduino (and the STM32F103) is that you wouldn't be able to batch up lots of data before writing as their RAM is so small (2kB or 20kB). so whilst the old STM32F103 would be able to read the ADC at a high rate, how could it be storing the data at the high rate with 20s intervals? I mean, if I'm using 7 channels, 20kB would get used up in ~0.2 s. Am I misunderstanding something here?

You're missing nothing! You are absolutely correct. You would either have to write more often, like every 50-100ms or so still might be feasible, or choose another STM32 with more RAM (e.g. the STM32F3/STM32F4 range), or go with that external RAM for buffering with all that entails. I can't think of any STM32s that have worse specs on their ADCs than the F103, except that some in the STM32F0 range might come with only one unit instead of two. #justsayin on the chance you were making a bunch and wanted to push the cost down a bit.

I'm planning on using a Teensy 3.6 anyway that has 256K (I think that means 256KB) of RAM so that should be able to write then dump to the micro SD card via the native port without a problem (I think).

That sounds fine. You'll have plenty of room to work with if you drop the write interval to about 1/2 second. How long did you intend to leave this device in operation for, and how critical is it that you not lose data? I'm getting a little concerned about burning out the flash from overuse, which could happen if you write the metadata block back too often. If you intend to run for months, you might think about pre-allocating a big file on the flash, using the file data section as a big ring buffer, and not writing to the metadata at all.

If I loose data I can just re-record it, it's not a huge issue. as for how much it's going to be used, it'll probably be used in ~1hr intervals a couple of times, not continuously for months. I don't really mind too much a $5 card burns out every couple of months anyway, they are fairly cheap.

I plan on progamming this on the Arduino IDE as it seems the most simple for a beginner, do you recommend this?

A new development is that apparently as I'm trying to amplify this signal from a strain gauge-wheatstone bridge setup there will be an issue using multiple channel inputs to the ADCs. apparently noise is increased as when using multiple channels you are multiplexing, I was unaware of this, let alone it being a problem. I guess I could just try 1 channel and then multiple channels and see if there is a difference. (you don't need to read this but the link to the discussion is here if you are interested: https://www.eevblog.com/forum/microcontrollers/getting-gain-on-adc-signals/ )

[rstofer]

On the back side of the Teensy 4, there appears to be a pad layout for an SD socket:
https://www.pjrc.com/store/teensy40.html

The Teensy 4 might be out of the power budget:  It requires 100 mA when running full speed.

Using the Arduino IDE would be my absolute next-to-last choice.  The only IDE that is worse is chisel and stone tablet.  OTOH, once TeensyDuino is installed, the Arduino IDE will sort out which toolchain to use and that's a handy feature.  It uses a particularly regrettable color scheme and, via Google, there are alternatives that don't display the error messages in invisible red-orange against black.  It also doesn't support the notion of multiple files in a hierarchy or tree structure.  That chisel looks better and better.

There is an add-in tool for Microsoft Visual Studio called Visual Micro that provides an identical Arduino capability with a more  professional IDE.  I don't know if it works with the TeensyDuino toolchain.  There is a difference between Microsoft Visual Studio and Microsoft Visual Studio Code - discussed after Eclipse.

I hesitate to recommend Eclipse simply because start-up configuration can be a challenge.  It's a nice IDE but it might be an advanced topic.

I have been using Visual Studio Code for a Lattice FPGA project and it's pretty nice.  There is an Arduino add-in but, again, I don't know if it works with TeensyDuino but it does allow for other ARM32 chips so I don't see why it wouldn't.  It requires the complete Arduino IDE and TeensyDuino is an add-on to that environment so it probably shows up in Visual Studio Code.  Do a test:  Add TeensyDuino to your Arduino installation, install Visual Studio Code (Community Edition) and then install the Arduino extension.  See how it all works out.  VS Code is a very advanced editor - way beyond my need, but it's pretty nice.

The only problem I have with VS Code is that it doesn't know how to print - anything.  One of the gurus pointed out that he had never needed to print his code for mark-up or review.  Well, goody for him!  I'm a mere mortal, I need printed copies.  Not a big deal, I can use Notepad++ to open the files and print them.  Actually, Notepad++ is a pretty nice editor and, while I haven't chased it down a rabbit hole, it appears it can be used with Arduino.  Google has some info.

Lots of choices and the Arduino IDE isn't in my selection set.

[rstofer]

Maybe something like this?

https://store.digilentinc.com/openlogger-high-resolution-portable-data-logger/

[powerfly]

An alternative that I think you mentioned at the end of your comment is to store maybe like 0.5Mb of data on the native RAM and whilst the other code is still reading data in, to dump that 0.5 Mb to a memory card, is that what you were saying?

More or less, reserving some space to buffer some of the filesystem metadata. Take care that you don't surpass your filesystem's limitations. You might find it easier on the whole to work in terms round wall-clock time, maybe do 2-second chunks instead, which is more than enough time to create or append a file on the flash.

how can you actually acheive this (code wise), how can I get the 10kHz aquistition rate not to slow down when dumping 512KB of data to an SD card?

[radiolistener]

powerfly, you can allocate two buffers. While first buffer is capturing, you can write second buffer to the sdcard. Then swap them and repeat.

[tooki]

Your Arduino mate was doing it wrong. He would have had better luck if he batched up a whole sector of data before writing it out, and I bet he was also formatting the output as text instead of binary.  (But the Arduino fans don't usually think at this level...)

Arduino’s raison d’être is to make it easy for beginners to, well, begin. It’s quite uncalled for to disparage beginners for not being experts in electronics and/or programming.