help with rapid ADC data aquizition

[powerfly]

 I am trying to obtain ADC voltage readings at a rate of 10kHz with a resolution of 12 bits or more. This is to be done with a portable, small device, such as a microcontroller with a small power bank attached to it. I would like to be able to use this with 7 data channels, and if possible even up to 24 channels. I have been looking at microcontrollers to do the job, but I'm stuggling to find a suitable one (or maybe I just need guidance on how to use them properly).

7 channels of 12 bit data is 7*1.5 bytes = 10.5 bytes.
10kHz --> 105kb/s of data storage.
Ideally, at least 20s of data could be stored at this rate before a short interval and then repeat.

1. Are there any microcontrollers that would be able to write at these sorts of speeds straight to an SD card or memory stick? - Someone I know tried doing this with an Arduino uno and the data logging was far too slow. would a teensy 3.6 work as it has a native micro sd card slot?

2. If this does not work, is it possible to store data at this rate to flash memory on a micro controller? apparently it is possible with arduinos, although they do not have enough flash memory. A teensy 4.0 might just be able to do the job, although I am not sure how to write at this rate to the flash memory of a teensy, it has a different architechture to most arduinos which I think makes things more tricky. Are there any other micro controllers I should be looking at?

3. Is a microcontroller the best option? I have thought about possibly using a rasberry pi, although I have no experience with theese and basically no coding experience, which might mean it is tricky. The 4 Gb of ram is appealing though and if the right sampling rates are attainable this could be a great solution. Due to the size and probably more tricky coding this option seems less appealing than a micro controller atm.

Bear in mind I have very limited experience with coding, any help would be much appreciated!

[Wimberleytech]

Silicon Labs sells 8-bit micros with 12-bit ADCs on them.  The C8051F020 supports a sample rate up to 100ksps.  In addition, it has an on-board 8 input mux.

[JimRemington]

would a teensy 3.6 work

Yes, but there are only two independent ADCs. A Teensy 3.2 will work too.

[jhpadjustable]

1. Just about any 32-bit MCU with an onboard ADC could do it. For perspective, an STM32F103 (bluepill, etc.) has a 1Msps ADC with 8 or more inputs depending on package. If zero skew and zero error isn't a requirement, you can sample as many channels as you like and pass them along to your storage device, maybe do a bit of oversampling on the way. You can even use DMA and timers to handle all the grunt work for you. Teensy 3.x would probably be enough power for the job.

2. SD cards are pretty easy. They can be used in an SPI-like mode, in case your chosen MCU doesn't have a proper SDIO peripheral or you don't want to use it for some reason. Beware that flash lifetime is rated by the number of writes. After you exceed the limit, you may find data isn't being written correctly. Maybe that's not a big problem if you have 8GB of SD card that does its own write leveling. Speed might still be a concern. You would need to qualify your particular choice of SD card. If you really don't need non-volatile storage, you could use a pseudo-SRAM with a serial interface, for example.

3. SD cards are pretty simple. USB host is hard. The trouble with the Pi is that the multitasking OS gets in the way, and it's also a bit anemic as to peripherals.

[powerfly]

would a teensy 3.6 work

Yes, but there are only two independent ADCs. A Teensy 3.2 will work too.

Are you saying that a teensy 3.6 would fully be able to do the job, or that it has ADCs?

I think the ADCs on the teensy each have several channels so it could be used for more than 2 channels couldn't it?

1. Just about any 32-bit MCU with an onboard ADC could do it. For perspective, an STM32F103 (bluepill, etc.) has a 1Msps ADC with 8 or more inputs depending on package. If zero skew and zero error isn't a requirement, you can sample as many channels as you like and pass them along to your storage device, maybe do a bit of oversampling on the way. You can even use DMA and timers to handle all the grunt work for you. Teensy 3.x would probably be enough power for the job.

2. SD cards are pretty easy. They can be used in an SPI-like mode, in case your chosen MCU doesn't have a proper SDIO peripheral or you don't want to use it for some reason. Beware that flash lifetime is rated by the number of writes. After you exceed the limit, you may find data isn't being written correctly. Maybe that's not a big problem if you have 8GB of SD card that does its own write leveling. Speed might still be a concern. You would need to qualify your particular choice of SD card. If you really don't need non-volatile storage, you could use a pseudo-SRAM with a serial interface, for example.

3. SD cards are pretty simple. USB host is hard. The trouble with the Pi is that the multitasking OS gets in the way, and it's also a bit anemic as to peripherals.

Thank you for your in depth response, I think it is helping me figure things out

At the moment I'm going for a teensy 3.6 or 4.0, it sounds from your comment that you are reccomending a micro-controller over a rasberry Pi.

I'm not quite understanding what you are saying about micro-SD cards. I am happy to spend a bit of money on a decent micro-SD card if it will store the data at the rate I need, for this a teensy 3.6 might be ideal as it has a native micro-SD port. Are there any micro-SD cards that could be used in conjunction with a teensy (or other microcontroller) and get the desired data rate? If so, what kind of Micro SD card?

As for the volatile RAM thing - Is it simple to add on this 64mb of ram to a teensy, temporarily store data there and subsequently dump the memory to a card when a chunk of data has been collected? If so, it does not matter that it is volatile.

[radiolistener]

7 channels of 12 bit data is 7*1.5 bytes = 10.5 bytes.

May be MAX197 will fits your needs.
There is a module on aliexpress: https://www.aliexpress.com/item/32952117207.html
Didn't tried it, but looks good.

[jhpadjustable]

Pi-type SBCs don't generally have decent ADCs onboard. You'd have to use an external ADC or seven, and then deal with multiplexing among all the ports, initiating conversions on a tightly timed basis, among other things. A Pi just isn't going to be able to do all that very reliably while Linux is running, managing a filesystem and block device, etc. So the Teensy 3.6 is a good choice if the on-board ADCs are up to it.

Your data rates are on the order of compressed lossy video recording, which isn't a very demanding requirement at all. I doubt any but the absolute cheapest boot-sale cards would have trouble. On the other hand, the cards actually contain a small MCU that does the wear-leveling, serial protocol handling, and so forth, and that code is a bit of a wildcard. Testing is the only way to be sure. You can get some idea by copying a very large file to it through a USB card reader. As a guess, if you can get 300kB/sec into it (200% margin, or 100% margin if you spare yourself the effort of packing the extra 4 bits out of each word), you probably won't have any trouble in the final application. Do beware of OS caches giving you falsely favorable results when you test. Or just buy Sandisk and not worry

The RAM I pointed out is just an SPI device. You use it more or less the same as you would an SPI flash. But you've got more than enough memory on board the Teensy to buffer for a few seconds, maybe even enough to hot-swap the card during operation if you're quick about it, and it takes some careful software design to work with that external RAM.

[powerfly]

Pi-type SBCs don't generally have decent ADCs onboard. You'd have to use an external ADC or seven, and then deal with multiplexing among all the ports, initiating conversions on a tightly timed basis, among other things. A Pi just isn't going to be able to do all that very reliably while Linux is running, managing a filesystem and block device, etc. So the Teensy 3.6 is a good choice if the on-board ADCs are up to it.

Your data rates are on the order of compressed lossy video recording, which isn't a very demanding requirement at all. I doubt any but the absolute cheapest boot-sale cards would have trouble. On the other hand, the cards actually contain a small MCU that does the wear-leveling, serial protocol handling, and so forth, and that code is a bit of a wildcard. Testing is the only way to be sure. You can get some idea by copying a very large file to it through a USB card reader. As a guess, if you can get 300kB/sec into it (200% margin, or 100% margin if you spare yourself the effort of packing the extra 4 bits out of each word), you probably won't have any trouble in the final application. Do beware of OS caches giving you falsely favorable results when you test. Or just buy Sandisk and not worry

The RAM I pointed out is just an SPI device. You use it more or less the same as you would an SPI flash. But you've got more than enough memory on board the Teensy to buffer for a few seconds, maybe even enough to hot-swap the card during operation if you're quick about it, and it takes some careful software design to work with that external RAM.

Thanks, your answer has provided more insight yet again.

It sounds like the external RAM can be complicated and as I'm a begginer I'll try and avoid that.

I know a guy who tried reading to an SD card using an Arduino uno and a shield that had a memory card slot on it. He said that there was an issue because it's not simply transferring data so the write speed isn't the main limiting factor. He said that the uno has to open a file, store it on the card, close it, then write another file or something. because of this it could only store a reading (12 bits) every 2ms or so which is far too slow for the rate I'm trying to achieve. If this is the only way to do it, then the write speed isn't the rate limiting step and is almost irrelevant.

Is this just an issue with not coding correctly and there is a way just to keep a file open and steam all of your data into it for like 20s before closing that file and making a new one?

An alternative that I think you mentioned at the end of your comment is to store maybe like 0.5Mb of data on the native RAM and whilst the other code is still reading data in, to dump that 0.5 Mb to a memory card, is that what you were saying?

[ a guess a card like this is the best thing: https://www.amazon.co.uk/SanDisk-microSDHC-Memory-Adapter-Performance/dp/B073JWXGNT/ref=sr_1_4?adgrpid=52894980363&gclid=CjwKCAiAuK3vBRBOEiwA1IMhum67ArXvd2iUwrEJ6WjaHgV0Qy4URl70Pw1gM7EKH2hPYCLI53zoiBoCg7kQAvD_BwE&hvadid=259084611967&hvdev=c&hvlocphy=1006976&hvnetw=g&hvpos=1t1&hvqmt=e&hvrand=9654974697421066106&hvtargid=kwd-295891704216&hydadcr=28148_1752695&keywords=micro+sd+card&qid=1575712020&sr=8-4 it claims up to 98mb/s write speed]

[radiolistener]

you can do in on stm32 microcontroller. But the type of controller depends on your needs. If you're needs just to get date from adc and write it to sdcard, you can do it on a cheap two-dollar stm32 microcontroller. And you can use integrated adc.

[powerfly]

you can do in on stm32 microcontroller. But the type of controller depends on your needs. If you're needs just to get date from adc and write it to sdcard, you can do it on a cheap two-dollar stm32 microcontroller. And you can use integrated adc.

Can do what? I'm not exactly sure what you're replying to, if it's one of the questions in my post above, which one?

cheap integrated ADCs aren't usually 12 bit and are often not 10kHz either.

[Wimberleytech]

you can do in on stm32 microcontroller. But the type of controller depends on your needs. If you're needs just to get date from adc and write it to sdcard, you can do it on a cheap two-dollar stm32 microcontroller. And you can use integrated adc.

Can do what? I'm not exactly sure what you're replying to, if it's one of the questions in my post above, which one?

cheap integrated ADCs aren't usually 12 bit and are often not 10kHz either.

Restating my earlier post...the Silabs ADC IS 12 bits and 100ksps!

[jhpadjustable]

Your Arduino mate wasn't quite up to the challenge. He might have had better luck if he batched up a whole sector of data before writing it out, and writing binary instead of ASCII text. Still not bad for a single-threaded approach. I do maintain that you skip the bit-packing and just store your values in aligned 16-bit units, especially if you've got gigabytes of flash to play with, so that you'll never have to worry about alignment on the reading end or the writing end.

An alternative that I think you mentioned at the end of your comment is to store maybe like 0.5Mb of data on the native RAM and whilst the other code is still reading data in, to dump that 0.5 Mb to a memory card, is that what you were saying?

More or less, reserving some space to buffer some of the filesystem metadata. Take care that you don't surpass your filesystem's limitations. You might find it easier on the whole to work in terms round wall-clock time, maybe do 2-second chunks instead, which is more than enough time to create or append a file on the flash.

That card is great, no worries.

cheap integrated ADCs aren't usually 12 bit and are often not 10kHz either.

Welcome to the 32-bit MCU world.   Again, the hoary old STM32F103 has two 1Msps ADCs with a headline spec of 12 bits, which you could oversample by 8x (or 16x if you use both) in this application. Other, newer STM32 members are even better equipped, in case that's not enough margin for you.

[powerfly]

Your Arduino mate was doing it wrong. He would have had better luck if he batched up a whole sector of data before writing it out, and I bet he was also formatting the output as text instead of binary.  (But the Arduino fans don't usually think at this level...) I still recommend that you skip the bit-packing and just store your values in aligned 16-bit units, especially if you've got gigabytes of flash to play with, so that you'll never have to worry about alignment on the reading end or the writing end.

An alternative that I think you mentioned at the end of your comment is to store maybe like 0.5Mb of data on the native RAM and whilst the other code is still reading data in, to dump that 0.5 Mb to a memory card, is that what you were saying?

More or less, reserving some space to buffer some of the filesystem metadata. Take care that you don't surpass your filesystem's limitations. You might find it easier on the whole to work in terms round wall-clock time, maybe do 2-second chunks instead, which is more than enough time to create or append a file on the flash.

That card is great, no worries.

cheap integrated ADCs aren't usually 12 bit and are often not 10kHz either.

Welcome to the 32-bit MCU world.   Again, the hoary old STM32F103 has two 1Msps ADCs with a headline spec of 12 bits, which you could oversample by 8x (or 16x if you use both) in this application. Other, newer STM32 members are even better equipped, in case that's not enough margin for you.

Thanks, that sounds good. the thing with the arduino (and the STM32F103) is that you wouldn't be able to batch up lots of data before writing as their RAM is so small (2kB or 20kB). so whilst the old STM32F103 would be able to read the ADC at a high rate, how could it be storing the data at the high rate with 20s intervals? I mean, if I'm using 7 channels, 20kB would get used up in ~0.2 s. Am I misunderstanding something here?

I'm planning on using a Teensy 3.6 anyway that has 256K (I think that means 256KB) of RAM so that should be able to write then dump to the micro SD card via the native port without a problem (I think).

Also, if you could point me in the right direction regarding how to do all the coding, that would be super helpful as I'm a coding noob. I think I'd like to use the arduino IDE if possible as it seems simplest

[rstofer]

In theory, you can transfer multiple megabytes per second to an SD card depending on the model
https://www.expertreviews.co.uk/storage/1404380/how-to-choose-an-sd-card-class-and-speed-ratings-explained

But you probably need something faster than an Arduino to do it and the code probably needs to be tightened up A LOT!  DMA and multiple buffers will help.

Horsepower matters!  How about the Teensy 4?  It is blazing fast (600 MHz) has two 12-bit ADCs along with 14 analog pins.  This doesn't really matter because the ADCs could be external on an SPI or I2C bus and run through the DMA channel.  Given a bunch of internal memory, creating packets for the SD card will be very fast.

https://www.pjrc.com/store/teensy40.html

Not that I would necessarily go there but there is an Arduino compatible library for the chip called TeensyDuino which can be installed under the Arduino IDE although it escapes me why anyone would voluntarily use that IDE.  But the library installation includes the proper toolchain for the chip (ARM, not ATmega, obviously).  Keep the library, moved to anything else for the IDE.

The ARM chip itself:
https://www.nxp.com/docs/en/nxp/data-sheets/IMXRT1060CEC.pdf

[powerfly]

In theory, you can transfer multiple megabytes per second to an SD card depending on the model
https://www.expertreviews.co.uk/storage/1404380/how-to-choose-an-sd-card-class-and-speed-ratings-explained

But you probably need something faster than an Arduino to do it and the code probably needs to be tightened up A LOT!  DMA and multiple buffers will help.

Horsepower matters!  How about the Teensy 4?  It is blazing fast (600 MHz) has two 12-bit ADCs along with 14 analog pins.  This doesn't really matter because the ADCs could be external on an SPI or I2C bus and run through the DMA channel.  Given a bunch of internal memory, creating packets for the SD card will be very fast.

https://www.pjrc.com/store/teensy40.html

Not that I would necessarily go there but there is an Arduino compatible library for the chip called TeensyDuino which can be installed under the Arduino IDE although it escapes me why anyone would voluntarily use that IDE.  But the library installation includes the proper toolchain for the chip (ARM, not ATmega, obviously).  Keep the library, moved to anything else for the IDE.

The ARM chip itself:
https://www.nxp.com/docs/en/nxp/data-sheets/IMXRT1060CEC.pdf

I have been discussing using either the teensy 3.6 or 4.0 (see above comments). the advantage of the 3.6 is that it has the built in SD card reader and the processor is probably powerful enough. The 4.0 does have the added power and RAM, and I guess adding a card reader wouldn't be too difficult (although rapidly storing data to the card may be more difficult?)

When you say the code needs to tightened up a lot, how would one go about doing this?

jhpadjustable mentioned some stufff he thinks will help, although having someone point me in the right direction of how to do this would be helpful. I am very much a beginner when it comes to coding. the teensyduino library is appealing as it seems like it would be simple to code in.

[jhpadjustable]

Thanks, that sounds good. the thing with the arduino (and the STM32F103) is that you wouldn't be able to batch up lots of data before writing as their RAM is so small (2kB or 20kB). so whilst the old STM32F103 would be able to read the ADC at a high rate, how could it be storing the data at the high rate with 20s intervals? I mean, if I'm using 7 channels, 20kB would get used up in ~0.2 s. Am I misunderstanding something here?

You're missing nothing! You are absolutely correct. You would either have to write more often, like every 50-100ms or so still might be feasible, or choose another STM32 with more RAM (e.g. the STM32F3/STM32F4 range), or go with that external RAM for buffering with all that entails. I can't think of any STM32s that have worse specs on their ADCs than the F103, except that some in the STM32F0 range might come with only one unit instead of two. #justsayin on the chance you were making a bunch and wanted to push the cost down a bit.

I'm planning on using a Teensy 3.6 anyway that has 256K (I think that means 256KB) of RAM so that should be able to write then dump to the micro SD card via the native port without a problem (I think).

That sounds fine. You'll have plenty of room to work with if you drop the write interval to about 1/2 second. How long did you intend to leave this device in operation for, and how critical is it that you not lose data? I'm getting a little concerned about burning out the flash from overuse, which could happen if you write the metadata block back too often. If you intend to run for months, you might think about pre-allocating a big file on the flash, using the file data section as a big ring buffer, and not writing to the metadata at all.

This doesn't really matter because the ADCs could be external on an SPI or I2C bus and run through the DMA channel.

Why, they're free with the chip.

[radiolistener]

1. Are there any microcontrollers that would be able to write at these sorts of speeds straight to an SD card or memory stick?

yes, you can do it with STM32 microcontrollers. But they have different performance and different peripherals, so you're needs to choice right type of STM32 microcontroller.

3. Is a microcontroller the best option?

yes

[radiolistener]

cheap integrated ADCs aren't usually 12 bit and are often not 10kHz either.

cheap STM32 microcontrollers have 1-2 ADC with 12 bit resolution which can work with sample rate up to 1 MHz. And about 8 channels with analog multiplexer.

That's enough to capture all 8 channels with 10 kHz sample rate and write it to sdcard. Such solution will needs to share 1-2 ADC between 8 channels. For example you can use first ADC to digitize odd channels and second ADC to digitize even channels.

[rstofer]

In theory, you can transfer multiple megabytes per second to an SD card depending on the model
https://www.expertreviews.co.uk/storage/1404380/how-to-choose-an-sd-card-class-and-speed-ratings-explained

But you probably need something faster than an Arduino to do it and the code probably needs to be tightened up A LOT!  DMA and multiple buffers will help.

Horsepower matters!  How about the Teensy 4?  It is blazing fast (600 MHz) has two 12-bit ADCs along with 14 analog pins.  This doesn't really matter because the ADCs could be external on an SPI or I2C bus and run through the DMA channel.  Given a bunch of internal memory, creating packets for the SD card will be very fast.

https://www.pjrc.com/store/teensy40.html

Not that I would necessarily go there but there is an Arduino compatible library for the chip called TeensyDuino which can be installed under the Arduino IDE although it escapes me why anyone would voluntarily use that IDE.  But the library installation includes the proper toolchain for the chip (ARM, not ATmega, obviously).  Keep the library, moved to anything else for the IDE.

The ARM chip itself:
https://www.nxp.com/docs/en/nxp/data-sheets/IMXRT1060CEC.pdf

I have been discussing using either the teensy 3.6 or 4.0 (see above comments). the advantage of the 3.6 is that it has the built in SD card reader and the processor is probably powerful enough. The 4.0 does have the added power and RAM, and I guess adding a card reader wouldn't be too difficult (although rapidly storing data to the card may be more difficult?)

Have you looked at the Horsepower Chart - the 4.0 is more than 5 times as fast as the 3.6
https://www.pjrc.com/store/teensy40.html
Speed is good!

When you say the code needs to tightened up a lot, how would one go about doing this?

I haven't looked closely at the TeensyDuino libraries but the stuff for the ATmega chip doesn't use any features like DMA or fancy interrupt drivers.  Mostly because the chip doesn't have much.

You would want to be able to concurrently transfer values from DMA buffers to the SD card.  In a perfect world, the buffers would be exactly the size of a disk block (512 bytes) and every time the DMA filled one it would generate an interrupt to tell the SD write code to transfer more data to the card.

I don't know if the TeensyDuino library has most of this code.  It could because the 3.6 code might be quite similar but I wouldn't necessarily count on library code if I want raw speed over portability across platforms.  Bare metal programming comes to mind.

jhpadjustable mentioned some stufff he thinks will help, although having someone point me in the right direction of how to do this would be helpful. I am very much a beginner when it comes to coding. the teensyduino library is appealing as it seems like it would be simple to code in.

I do have a couple of Teensy 4.0 boards but I haven't played with them yet.  Some years back I used the RawHID code and a early Teensy to add switches, knobs and dials to Microsoft Flight Simulator using USB HID.  The code worked well so I have always been a fan of the Teensy products.

Sometimes, internal ADCs don't have the required performance and that's why I said it might not matter because you could use the 3 internal SPI buses (with 16 byte FIFOs) to connect to external ADCs with better performance.  I haven't looked to see that the SPI gadget connects through DMA but I'll bet it does.  So, basically, all of the ADC reading and transfer can be done in the background.  When a buffer gets filled, the code sets a flag to tell the mainline to  transfer the buffer to the SD card.  This too should be done by DMA so very little code execution is required.

There are many examples of using SD cards with one of the common FAT filesystems with the entire project running under FreeRTOS and while this is an advanced topic, it truly simplifies the code by cleaning up the inter-communications between reading ADCs and writing the SD.  Semaphores and queues (or buffer) plus some handshake signals control everything.

But, given an excess of horsepower, start simple and see how fast you can go.  If it meets your needs, there is no reason to dive deeper into the code.

However, having an SD adapter onboard makes the 3.6 very compelling.  I might start there and if I need more horsepower, I know where to get it.

[powerfly]

Thanks, that sounds good. the thing with the arduino (and the STM32F103) is that you wouldn't be able to batch up lots of data before writing as their RAM is so small (2kB or 20kB). so whilst the old STM32F103 would be able to read the ADC at a high rate, how could it be storing the data at the high rate with 20s intervals? I mean, if I'm using 7 channels, 20kB would get used up in ~0.2 s. Am I misunderstanding something here?

You're missing nothing! You are absolutely correct. You would either have to write more often, like every 50-100ms or so still might be feasible, or choose another STM32 with more RAM (e.g. the STM32F3/STM32F4 range), or go with that external RAM for buffering with all that entails. I can't think of any STM32s that have worse specs on their ADCs than the F103, except that some in the STM32F0 range might come with only one unit instead of two. #justsayin on the chance you were making a bunch and wanted to push the cost down a bit.

I'm planning on using a Teensy 3.6 anyway that has 256K (I think that means 256KB) of RAM so that should be able to write then dump to the micro SD card via the native port without a problem (I think).

That sounds fine. You'll have plenty of room to work with if you drop the write interval to about 1/2 second. How long did you intend to leave this device in operation for, and how critical is it that you not lose data? I'm getting a little concerned about burning out the flash from overuse, which could happen if you write the metadata block back too often. If you intend to run for months, you might think about pre-allocating a big file on the flash, using the file data section as a big ring buffer, and not writing to the metadata at all.

If I loose data I can just re-record it, it's not a huge issue. as for how much it's going to be used, it'll probably be used in ~1hr intervals a couple of times, not continuously for months. I don't really mind too much a $5 card burns out every couple of months anyway, they are fairly cheap.

I plan on progamming this on the Arduino IDE as it seems the most simple for a beginner, do you recommend this?

A new development is that apparently as I'm trying to amplify this signal from a strain gauge-wheatstone bridge setup there will be an issue using multiple channel inputs to the ADCs. apparently noise is increased as when using multiple channels you are multiplexing, I was unaware of this, let alone it being a problem. I guess I could just try 1 channel and then multiple channels and see if there is a difference. (you don't need to read this but the link to the discussion is here if you are interested: https://www.eevblog.com/forum/microcontrollers/getting-gain-on-adc-signals/ )

[rstofer]

On the back side of the Teensy 4, there appears to be a pad layout for an SD socket:
https://www.pjrc.com/store/teensy40.html

The Teensy 4 might be out of the power budget:  It requires 100 mA when running full speed.

Using the Arduino IDE would be my absolute next-to-last choice.  The only IDE that is worse is chisel and stone tablet.  OTOH, once TeensyDuino is installed, the Arduino IDE will sort out which toolchain to use and that's a handy feature.  It uses a particularly regrettable color scheme and, via Google, there are alternatives that don't display the error messages in invisible red-orange against black.  It also doesn't support the notion of multiple files in a hierarchy or tree structure.  That chisel looks better and better.

There is an add-in tool for Microsoft Visual Studio called Visual Micro that provides an identical Arduino capability with a more  professional IDE.  I don't know if it works with the TeensyDuino toolchain.  There is a difference between Microsoft Visual Studio and Microsoft Visual Studio Code - discussed after Eclipse.

I hesitate to recommend Eclipse simply because start-up configuration can be a challenge.  It's a nice IDE but it might be an advanced topic.

I have been using Visual Studio Code for a Lattice FPGA project and it's pretty nice.  There is an Arduino add-in but, again, I don't know if it works with TeensyDuino but it does allow for other ARM32 chips so I don't see why it wouldn't.  It requires the complete Arduino IDE and TeensyDuino is an add-on to that environment so it probably shows up in Visual Studio Code.  Do a test:  Add TeensyDuino to your Arduino installation, install Visual Studio Code (Community Edition) and then install the Arduino extension.  See how it all works out.  VS Code is a very advanced editor - way beyond my need, but it's pretty nice.

The only problem I have with VS Code is that it doesn't know how to print - anything.  One of the gurus pointed out that he had never needed to print his code for mark-up or review.  Well, goody for him!  I'm a mere mortal, I need printed copies.  Not a big deal, I can use Notepad++ to open the files and print them.  Actually, Notepad++ is a pretty nice editor and, while I haven't chased it down a rabbit hole, it appears it can be used with Arduino.  Google has some info.

Lots of choices and the Arduino IDE isn't in my selection set.

[rstofer]

Maybe something like this?

https://store.digilentinc.com/openlogger-high-resolution-portable-data-logger/

[powerfly]

An alternative that I think you mentioned at the end of your comment is to store maybe like 0.5Mb of data on the native RAM and whilst the other code is still reading data in, to dump that 0.5 Mb to a memory card, is that what you were saying?

More or less, reserving some space to buffer some of the filesystem metadata. Take care that you don't surpass your filesystem's limitations. You might find it easier on the whole to work in terms round wall-clock time, maybe do 2-second chunks instead, which is more than enough time to create or append a file on the flash.

how can you actually acheive this (code wise), how can I get the 10kHz aquistition rate not to slow down when dumping 512KB of data to an SD card?

[radiolistener]

powerfly, you can allocate two buffers. While first buffer is capturing, you can write second buffer to the sdcard. Then swap them and repeat.

[tooki]

Your Arduino mate was doing it wrong. He would have had better luck if he batched up a whole sector of data before writing it out, and I bet he was also formatting the output as text instead of binary.  (But the Arduino fans don't usually think at this level...)

Arduino’s raison d’être is to make it easy for beginners to, well, begin. It’s quite uncalled for to disparage beginners for not being experts in electronics and/or programming.

[rstofer]

An alternative that I think you mentioned at the end of your comment is to store maybe like 0.5Mb of data on the native RAM and whilst the other code is still reading data in, to dump that 0.5 Mb to a memory card, is that what you were saying?

More or less, reserving some space to buffer some of the filesystem metadata. Take care that you don't surpass your filesystem's limitations. You might find it easier on the whole to work in terms round wall-clock time, maybe do 2-second chunks instead, which is more than enough time to create or append a file on the flash.

how can you actually acheive this (code wise), how can I get the 10kHz aquistition rate not to slow down when dumping 512KB of data to an SD card?

That's the whole point of using DMA channels and multiple buffers.  Once an SD operation is set up, it happens by automagic and generates an interrupt or sets a semaphore when complete.  Maybe there is a counting semaphore that, when non-zero, causes more buffers to be transferred until the semaphore reaches 0.  Maybe there is a circular queue of 512 byte sector data.  This operation wants to have very low overhead but needs to be elastic in terms of timing.

On the other end, there is a timer or something generating ADC start signals and, just guessing, the ADC values will fill up 512 byte buffers in the circular queue.  Again, these samples can be moved into the processor by DMA.  Some DMA channels are pretty sophisticated in how they can use circular queues.

What is really necessary, as a first step, is to understand the DMA controller.  Read the datasheet while imaging how the data will flow through the system.  You have to assume, and prove, that you can write faster than you can read.  If this isn't true, you have the wrong chip.

ADC -> DMA -> Buffer -> DMA -> SD Card where 'buffer' means a circular queue of sector sized blocks.  You will want several such blocks and, given the 512K of RAM on the Teensy 4 (there are some strange memory limitations that I don't understand, yet), you might as well use it up.  Maybe 32 buffers is sufficient, maybe 64.  You need to study the write time of the SD card along with the incoming ADC rate to figure out how many buffers you need.  While a buffer (block) is being written to SD, it is unavailable to the ADC stream.  The ADC stream needs to be certain it isn't overrunning the SD write buffers.  Maybe the SD card will support multiple sector write commands - Compact Flash will.

All of this will likely work but the programming is far beyond what Arduino users are used to seeing.  I sort of doubt that moving the data in a superloop (void loop() is a 'superloop') will be satisfactory since there is no concept of multi-tasking.

[rstofer]

Depending on how you want to post-process the data, you may not need an actual file system.  You can simply write contiguous sectors, starting at 0 (or wherever) and continuing on in the address space.

Then, you can use a utility like Linux's 'dd' command to read the raw sectors from the SD card into a named file in the Linux file system.  I suspect this can be done in Windows using one of the raw read utilities. 

Using the SD card for raw data would be faster and less complex but you would need to write a sentinal block at the end of the dataset to mark the end.  Given the speed of the Teensy 4 something like the ChaN FatFs might be suitable but if you wind up using FreeRTOS, there is a FS ready to go
https://www.freertos.org/FreeRTOS-Plus/FreeRTOS_Plus_FAT/index.html

A filesystem would be nice as long as the system can keep up.  Given DMA input and DMA output, the only thing the CPU does is handle the filesystem.  It should work fine.

[jhpadjustable]

how can you actually acheive this (code wise), how can I get the 10kHz aquistition rate not to slow down when dumping 512KB of data to an SD card?

I would not advise using Arduino for this. Its general spirit is to be simple and very transparent, to the point you can virtually reach out and touch every bit of data that you're working with, in order to aid understanding of programming and to get simple things done easily and quickly. In this case, to meet your brief, you don't want to handle every piece of data. You need to work at arm's length, on the "control plane", and let engines outside of the code push data around on the "data plane" while you sit back and steer.

Speaking for STM32, ST's Cube tools offer a way to pointy-clicky set that up quickly, leaving you to fill in a few hundred lines of code at most, and it does a decent job of that. NXP Processor Expert seems to fill that role for Kinetis. You may be able to bend the sample code found here to your needs: https://community.nxp.com/docs/DOC-96507  NXP app notes AN4639 and AN4590 seem to be useful reading as well. (I'm not a Kinetis guy, but it does seem like the STM32 hardware is a bit more suited to handling just this sort of data acquisition task.)

Arduino’s raison d’être is to make it easy for beginners to, well, begin. It’s quite uncalled for to disparage beginners for not being experts in electronics and/or programming.

It's a fair cop. I've toned it down.

Depending on how you want to post-process the data, you may not need an actual file system.  You can simply write contiguous sectors, starting at 0 (or wherever) and continuing on in the address space.

For the best of both worlds, set up a Potemkin FAT file system with one giant file covering the entire data space, and treat it as a fixed header that you need not touch. You then use direct block access with an offset to address the contents of the file. The output size seems to be upper-bounded to a couple of GB (504 million bytes/hour, a few hours at a session) and the card size seems to be lower-bounded to at least double that. So simple, compact FAT16 would suffice.

I've crafted a FAT16 image in the attached 7zip archive. When written to the SD card using dd, Etcher or other similar tool, there will appear one 2GB FAT16 partition containing a single large file DATA.BIN with 4 hours (2,016,000,000 bytes) in consecutive blocks, and some blocks and directory entries left over for another device to add post-session notes or whatever you will. Measured from the very beginning of the card, the user content area begins at byte address 1937168 (decimal), absolute sector #3783 (decimal, 512-byte basis), and extends for (3937500) 512-byte sectors. That's one less foray into the weeds for powerfly to have to deal with.

(Ideas for a stretch goal: the FAT tables themselves are mostly just consecutive uint16_t numbers which could be easily resynthesized with a for loop or two. If they were stripped out and reconstructed at card build time, the few sectors remaining could be converted to C arrays to occupy a small amount of flash, and the recording device could "format" any card on the fly. If there were enough flash space available, it might (or might not) be easier and less in-the-weeds to bring along something like zlib and just decompress directly to the SD card's blocks.)

Using the SD card for raw data would be faster and less complex but you would need to write a sentinal block at the end of the dataset to mark the end.

A block "timestamp" or counter would maintain append-only semantics, which eliminates a little bit of uncertainty in timing which the card and any filesystem code may or may not introduce due to rewriting the same block twice. If one chooses a sufficiently unique starting number (the time from a real-time clock would be perfect, but a good randomness source such as user input would work too), and assuming no wrapping, you'll know you've reached the end of the recording when you see any out-of-sequence block serial number, fortunately including 0 or -1 from unused flash.

[mikerj]

Your Arduino mate was doing it wrong. He would have had better luck if he batched up a whole sector of data before writing it out, and I bet he was also formatting the output as text instead of binary.  (But the Arduino fans don't usually think at this level...)

Arduino’s raison d’être is to make it easy for beginners to, well, begin. It’s quite uncalled for to disparage beginners for not being experts in electronics and/or programming.

Arduino makes it easy and fast to do things the wrong way.  OTOH replace the Arduino ecosystem with a conventional C compiler and decent editor, and the boards and shields are cheap and useful for all sorts of things.

[Nominal Animal]

how can you actually acheive this (code wise), how can I get the 10kHz aquistition rate not to slow down when dumping 512KB of data to an SD card?

That's the whole point of using DMA channels and multiple buffers.  Once an SD operation is set up, it happens by automagic and generates an interrupt or sets a semaphore when complete.  Maybe there is a counting semaphore that, when non-zero, causes more buffers to be transferred until the semaphore reaches 0.  Maybe there is a circular queue of 512 byte sector data.  This operation wants to have very low overhead but needs to be elastic in terms of timing.

If you do not use a filesystem (I wouldn't bother), I would have the acquisition only do say 500 samples per 512-byte sector, and at least a monotonic 32-bit counter (8 bytes) at the start of each block.  (A 32-bit counter suffices for roughly a terabyte.)

The data acquisition part generate all the 512 bytes per sector, just storing the incremented counter before the actual samples.  It does not "cost" more than a couple of variable assignments and increments in machine code (in C/C++ it is just an assignment), so it definitely should not affect the ADC sampling at all.

The trick here is that you "format" the SD storage by initializing the very first and last sectors, with the first sector having counter value 0, and the last sector counter value one less than the number of sectors. (This is one of the two special cases in this scheme, and means the very last sector was the most recently written one.  The other special case is when the counter rolls over; that's why a 32-bit counter suffices for only a terabyte or so, and not two (512×2³²=2,199,023,255,552).

Because the sectors are written in sequence, you get wear leveling for free.  A reader, and the microcontroller when starting up, reads the first and last sectors to determine where the most recently written sector is (often having to do a binary search -- but that's less than 32 additional sectors to read -- and only the counter parts of the sectors need to be read); and the sector following it is the next one to be replaced/written to.  (The method should be obvious, but if you don't see it, let me know and I or someone else here will describe the exact logic.  I'm trying to avoid writing too much of a wall of text here!)

In essence, this works very much like a super-lightweight, wear-leveling, continuous data stream storage system, with 508 bytes available per each 512-byte sector.

You will want several such blocks and, given the 512K of RAM on the Teensy 4 (there are some strange memory limitations that I don't understand, yet), you might as well use it up.

Very good point.  Teensy 4 has a slightly oddball architecture.  It has 1M of RAM, and 2M of Flash, but half the RAM is tightly coupled; so, essentially, you have two 512k RAM "banks" of sorts.  Usually, code is copied to the tightly coupled RAM to run faster.

This is such a new architecture, it is taking a bit of time for the Teensyduino support to stabilize and grow all the necessary features.  I see powerfly already started a thread at the PJRC forums, where KurtE and others point out the features still under development.

One crucial point I'd like to make, is to first define the terms and data rates precisely.
Rounding up the figures, let's say you need 10 channels, each sampling at 9-16 bits per sample, at a sample rate of 10,000 samples per second per channel, i.e. two bytes per sample, at a rate of 10×2×10,000 = 200,000 bytes per second.

As Defragster mentioned at the linked thread, the standard SdFat library (it's written by Paul Stoffregen, the developer of the Teensies) can do 5,000,000 - 15,000,000 bytes per second to an SD card (depending on the card).

So, in theory, you should be good for at least 20 channels, at 400,000 bytes per second, and not have issues with the data throughput.  (Note that even on a 3.2 the SD card interface is just the connector, there are no active components needed.  The SD Adaptor at PJRC.com is for the 8-bit Teensies, 2.0 and 2.0++, and has level shifting as they use 5V logic instead of the 3.3V logic like later Teensies and SD cards.)

Whether the internal ADC suffices for your case, or if you should use external ADCs, I can't really tell.  I've used both, but I don't really have the test equipment to say what kind of real-world results you can expect.  The common I2C ADC modules like ADS1x15, MCP342x, ADS1220 don't have the sample rate, but there are AD7606-based 8-channel 16-bit ADC modules capable of sampling at 200,000 samples/second; i.e. capable of sampling 8 channels at 25,000 samples/second.  The interfacing is not as easy as some other modules, but there are existing AD7606 Arduino projects, and you can get the modules from fleabay at 13-25 euros.  Since the sample rate is more than twice your required data rate, you should be able to use a single SPI bus (not the one connected to the SD card), to interleave samples from two separate boards, giving you 16 channels, 16 bits per sample, at 10,000 samples per second per channel.  Or, better yet, use two different SPI buses, and DMA to suck the interleaved samples from the AD7606 modules.

Personally, I would be tempted to go with the Teensy 4 and two AD7606 modules, because of the RAM and the speed available, but fact is, the software support isn't there yet.  For now, I'd say Teensy 3.5 or 3.6 is a better choice, and should have easily the oomph to do this (aside from whether the internal ADCs suffice or not).  I would probably also use 4096-byte sectors, just to reduce the (counter) overhead; you can probably use about 200k on the Teensy 3.5/3.6 for buffering sample data, or about 50 buffers, which should easily suffice even over SD card hiccups.

[rstofer]

I first mentioned the external ADCs and now that it has come up again, it's important to figure out if the sampling and transfers can be automagic.  If there needs to be a major exchange of commands just to transfer a single sample, DMA isn't going to work.  DMA wants a stream of data, not chit-chat.

It will be worth the time to figure out whether the internal ADCs are adequate because they likely can be connected through DMA.  Nevertheless, this needs to be proven.

In a perfect world, the foreground task wouldn't be involved with transfer at any stage.  It would simply set up the DMA registers and sit back and watch (for the next DMA register update).

This whole project is going to be built around data flow, not some superloop of code.

[tooki]

Arduino’s raison d’être is to make it easy for beginners to, well, begin. It’s quite uncalled for to disparage beginners for not being experts in electronics and/or programming.

It's a fair cop. I've toned it down.

Thank you!

Your Arduino mate was doing it wrong. He would have had better luck if he batched up a whole sector of data before writing it out, and I bet he was also formatting the output as text instead of binary.  (But the Arduino fans don't usually think at this level...)

Arduino’s raison d’être is to make it easy for beginners to, well, begin. It’s quite uncalled for to disparage beginners for not being experts in electronics and/or programming.

Arduino makes it easy and fast to do things the wrong way.  OTOH replace the Arduino ecosystem with a conventional C compiler and decent editor, and the boards and shields are cheap and useful for all sorts of things.

No doubt. But nonetheless, getting it done at all is still hugely valuable as a learning tool. I can say confidently that without Arduino, I would never, ever have even begun dabbling with MCUs, because as a non-programmer, I would never have been able to successfully interface with all the chips and components for which ready-made Arduino libraries exist. My programming skills are too basic to be able to implement interface timing, etc. Also, the Arduino ecosystem has produced hundreds of little breakout boards with (usually) decent minimal viable circuits. I can’t really say how many n00b mistakes I would have made if I’d had to try and get just the bare chips and stuff working. And finally, Arduino really brought down the cost of entry, given that historically, dev boards, EPROM programmers, IDEs, etc cost a lot of money. Would eBay be chock full of $2 ISP programmers if Arduino hadn’t come along? Maybe, but I doubt it.

So Arduino is fantastic for getting people into MCUs. And then, little by little, one can transition to using fewer premade components and libraries. (For example, I’m now starting to use bare Atmega chips instead of Arduino nanos, etc.) But many of us never will move past Arduino, simply because we aren’t programmers and don’t really aspire to be. And that’s OK. Don’t deny and begrudge us our successes just because it’s not the “optimal” way to solve a problem.

[Nominal Animal]

Excellent points, rstofer.

It looks like Teensy 3.2, 3.5, and 3.6 (but not LC or 4.0) have sufficiently accurate SAR ADCs (two of them!) to get 200,000 samples/second.  One does need to reduce the number of samples averaged (from the default 32) to 16 or less, but that should still leave at least 12 meaningful bits in the result, I think; I haven't verified this.

I don't think there is a way to DMA data from different analog input pins using a single ADC.  I could be wrong, though.

I would try a periodic interrupt that triggers the next conversion on the current ADC, then reads the conversion result from the other ADC, and selects the input pin for the next-next sample (the current ADC).  It has no waits or anything like that, so even a high-sounding rate like 200,000 interrupts per second should work okay.

I do not know if the overhead of such an interrupt is too much for SD card transfers.  The interrupt can only occur between byte transfers, so it is a matter of if the interrupt causes stretches a SPI clock cycle between bytes too long.

20 channels is pushing Teensy 3.2 to its limits (and half of those analog input pins are on pads underneath, which are a bit tricky to connect to), but might work fine on a 3.5 or 3.6, as they have a much higher clock rate, and the analog pins are easier to access.  I don't have a 3.5 or 3.6 myself to test, though.


I took a looksee at the AD7606 datasheet.

It turns out AD7606 can do 200 ksamples/second per channel, and can support both ±10V and ±5V analog voltage ranges when powered from 5V (AV_cc) while the logic voltage levels are 3.3V (V_drive).  Darned interesting to my needs as well; I wonder why I haven't encountered it before!  The modules at fleabay are suspiciously cheap compared to the chips themselves, although they look a lot like the circuit note.

It is completely controlled by logic-state pins (i.e., no command input, and only output is sample data; mode is set by tying control pins high or low), and is rather simple to operate.  It has three main modes: serial output, byte output/8-bit parallel output, and 16-bit parallel output.

To trigger a new sample (all channels are sampled simultaneously), you pull CONVST(s) high.  Within 45ns, the AD7606s pull BUSY high.  It will stay high for up to 4.2us (longer if oversampling is enabled).  When an AD7606 pulls BUSY back low, the samples are available.

To read the 8 samples in serial mode:

SCLK is high. The microcontroller pulls CS low for the AD7606, and waits 35ns or longer.  Then, it starts pulsing SCLK (down) 128 times at a maximum frequency of 15 MHz (and duty cycle between 40% and 60%, so this must be a single noninterrupted SPI transaction).  On each rising edge of SCLK, the microcontroller receives the next bit on the D_OUTA pin.  After the 128 bits, the microcontroller pulls CS high.

The byte mode is very similar:

RD and CS are high. The microcontroller pulls CS low, and RD low for at least 30ns, for the AD7606. On each rising edge of RD, the next 8 bits are available on DB7:0.  RD must be low for at least 30ns, and high for at least 15ns, so the max RD rate is about 20MHz at 3.3V V_DRIVE logic levels.  After the 16 bytes (rising edges of RD), the microcontroller pulls CS high.

The 16-bit parallel mode supports a linked CS and RD mode:

RD and CS are is high. The microcontroller pulls both low, and waits for at least 30ns (at 3.3V V_DRIVE), then reads the sample from DB15:0, and pulls both high again.  After at least 22ns, it can repeat this for the second channel; and repeat for all eight channels.

In the byte mode, an inline assembly routine on Teensy 3.2/3.5/3.6 can read the eight samples in about 1us per AD7606.

With three AD7606s/24 channels, and using an interrupt (the AD7606s can share the data lines, as long as only one has its CS low at any time), this is 30,000 interrupts per second, and 30ms or so of execution time, per second.  Not bad at all.  Whether the latencies caused by the 1us interrupts are too much jitter for the SD cards, I do not know.  (It is equivalent to about 25 cycles of the SD card SPI clock running at 25MHz, but only occurs between byte transfers, not mid-byte.)


So, it seems to me, trying out the built-in ADCs on a Teensy 3.5 or a 3.6 would be the way to go.
If they don't give good enough results, I'd go for the AD7606s, unless something better is found.

If the code to handle both the sampling and the SD card storage on a single Teensy is too complicated, then I'd use two, connecting the two using UART for commands, and SPI for sample data.  Then, the one storing the data to an SD card can definitely use DMA for the data (being SPI slave), and having enough RAM for buffering the data, can use the standard SdFat library to write the data to a standard FAT filesystem on the SD card, with a loop that just tries to keep up with the DMA.  (The DMA is connected to a completion interrupt, which sets up the next chunks, with at least a dozen or so such chunks in your buffer.)

Teensy 4.0 should be able to control three AD7606s in byte mode. The data pins would be 16-23 (DB0 pin 19, 18, 17, 16, 22, 23, 20, DB7 pin 21); pins 0 and 1 for UART; pins 10-13 for SPI; CONVST on pin 14 and RD on pin 15; and the three CSs, three BUSYs and RESET somewhere on pins 2-9.  There are an additional ten pins on the bottom of Teensy 4.0 on pads, but they're a bit tricky to solder to, but if used, would allow expanding to a fourth AD7606 for 32 channels. Teensy 4.0 would be the SPI master, although only providing CS, SCK, and MOSI/data out.  The CONVST pin would be a PWM or timer output, for optimal sampling stability while still allowing precise sample rate control.
The UART is used to set the sample rate and select the channels sampled (well, ALL are sampled, but only some are forwarded), with say a single-byte "OK" or "Error" acknowledgement (for simplicity and ease of use).

The nice thing is that if connected to USB, it could switch to USB mode, and since Teensy 4.0 has a 480Mbit/s USB interface, could provide up to the maximum 200,000 samples/second per channel for all 24 connected channels, making the same thing useful as a AD7606 data acquisition controller.

If my Christmas budget allows (I'm broke ), I'll get a couple of those AD7606 modules off fleabay, and try them with my Teensy 4.0.

[powerfly]

Just wow, I am surprised by the sheer number of replies from so many different members. Thank you so much for all being willing to help me along with my little project. I think I'm going to read the thread again as I definitely did not understand all of the comments above and some of it is a bit too much for me to take in just from reading it once.

My coding is very much beginner level so I'm not even sure what all of the terms discussed mean. If you could elaborate more on technical terms or provide links that describe them that would be helpful.

I'm really hoping that some pre-existing code is freely available as I've seen similar things for an Arduino:

https://forum.arduino.cc/index.php?topic=228549.0&fbclid=IwAR3U5bL_cAvS-P9N1w4Eq5ZJuPOiywBqPmLKXwgcUpS0Ki7DqYYJMuFkP50

https://github.com/greiman/SdFat/blob/master/examples/AnalogBinLogger/AnalogBinLogger.ino

In the links I've posted it's apparently possible to get 40 kHz 10 bit data to an uno, and by looking at the code (github link), I think a much higher rate is possible. The best option at the moment for my needs seems to be the T3.6, which I now have available to me (as well as a T4.0).

Does anyone know if there is an equivalent code to the one above but for ARM based processors such as those on the teensy 3.6? I currently don't have anyway near the coding capability to construct code like the one above and this project isn't primarily about coding, and I have a deadline for it. As the T3.6 has 23 analog in ports which are at least 13 bit, and at least initially I'll be using 8 pins or less it should be able to handle the 12bit 10kHz rate. It's just getting it done in the timeframe with my lack of coding experience may prove challenging.

Edit: I've been reading throught the thread, and looking up the definition of some of the terms and think I have a better idea of what they mean.

There a still many things I'm unsure of because of my lack of experience. I don't know how to set up a buffer, I don't know whether I should use a file system and if I do use one which one I should use etc.

Does the T3.6 have DMA channels built in, are they a software thing or are they external pieces of hardware?

As this is my first major coding project, it would be really helpful if someone knows of any script similar to the arduino one above. I don't even know where to start in constructing code with buffer systems, DMAs or anything similar.

[powerfly]

does anyone have experience with coding of that type? Or know of any existing available code of that type?

If that doesn't work I might just have to limit myself to maxing out the RAM and stopping and dumping that to a card and hope that that is good enough, as I'm not sure I have the time to learn how to make a code and then make a functioning code on how to make DMA buffers and stuff

[Nominal Animal]

powerfly, the reason you are getting such scattered responses is because you haven't specified the root problem.  Here are the things you should disclose FIRST:

• The input voltage range
• Reference type: absolute or ratiometric (to a reference or supply voltage)
• Required/desired precision, acceptable noise level
• Bandwidth (1 kHz?) or desired sample rate (10 kHz?) per channel
• Differential or single-ended measurement - common ground across all channels or not
• Number of channels
• Estimated duration of continued measurement

As revealed piecewise in this and the other thread, I too believe you need either a specialized ADC or at least an instrumentation amplifier per channel.  (Specialized ADCs like HX711 for strain gauges don't seem to have the sample rate needed, though.)

It seems to me you need a custom board to implement this, with at least an instrumentation amplifier (say, INA217) for each channel, connected to one or more ADCs (with single-ended inputs; the REF pins of the INA217s are connected to the ADC ground).  The INA217's are not difficult to layout, as each one needs just two 0.1uF/100nF bypass capacitors and a gain resistor.  The resistor precision and value is important, as it sets the gain, but the values are described in the datasheet.  The desired gain depends on the measured voltage range and the optimum range for the ADCs.

This is such a specialized thing that you have very little hope of finding an existing project and just tweak it to suit your purposes.

It is very likely, but not guaranteed, that you can use a Teensy 3.5 or 3.6's ADCs with a carrier board with the instrumentation amplifiers, and perhaps a couple of buttons to start and stop the measurements, and a small cheap I²C OLED display off fleabay to display the status (and maybe the file name pattern the data is being stored to).  The 128x32 and 128x64 modules are easy to use, nicely readable even in normal light conditions, but need to be protected from water; I especially like the white ones.  There are a lot of fonts etc. to choose from, too.

(The OLEDs are electrically noisy, so I'd have it well away from the amplifiers, and have extra filtering on its +VCC supply rail.  If the module has a 3.3V regulator, use the +5V VUSB power rail, as Teensies have pretty good filtering from that, including a ferrite bead.  Otherwise, use the 3.3V output from the Teensy, but with 100nF and 1uF decoupling capacitors, maybe even a ferrite bead.)


It looks like I was wrong, earlier!  It is possible to use DMA to sample from different analog input pins.

It involves using a Timer/PWM modules (TPM) or a Programmable Delay Block (PDB) to trigger the conversion at a stable rate with minimal sampling jitter, plus two DMA channels per ADC unit used.  The TPM or PDB triggers each ADC conversion at the total sample rate, with very little timing jitter.  The completion of the conversion triggers an 16-bit DMA transfer to a circular buffer, so the samples from one ADC unit are interleaved.  The second DMA channel is slaved to the first, so that it triggers immediately after the first one, and uses a small circular 32-bit buffer to transfer the sample channel (ADCx_SC1A, see pin2sc1a array for pin to sample mapping) to the ADC SC1A register.  You don't even need a DMA completion interrupt, because the offset in the buffer to the next sample to be stored is available as the TCD->DOFF member of the DMAChannel objects; your main loop simply examines those using a volatile access to see where the DMA will write the next sample.

Teensy 3.2, 3.5, and 3.6 have two ADC units.  Teensy 3.2 and 3.5 have 16 DMA channels, and 3.6 has 32 DMA channels, so this is not a problem at all.
I like the idea of using the two TPM modules, even though it restricts PWM outputs a lot, but then the total sample rate can be very precisely controlled.  (At 10 channels per ADC unit, and 10,000 samples per channel per second, the trigger interval is 1/200,000 seconds, or 5us; and thus the rate 200,000 Hz or 200 kHz.

Because the ADC and DMA handle the sampling, and you don't even need an ISR at all, you can safely use the standard SD card libraries to write the data to one or more files; the samples simply "appear" into SRAM completely without processor intervention, without using any processor time at all.  (There may be additional clock cycles inserted into instructions when both the processor and the DMA access the same memory, but I haven't checked.  It will not affect anything, including the SD card write routines, though, because they use the SPI subsystem and a couple of clock cycles per byte send via SPI in those routines should not affect anything at all.  For best results, let the DMA run at least a few samples ahead all the time.)

Pedro Pascal's (pedvide) ADC library implements some of these concepts.  The Teensyduino DMAChannel.h and DMAChannel.cpp describes how the DMA channels are manipulated, and provides a Teensy-portable abstraction for using them.

This looks like a very interesting, and tricky programming problem; just up my alley.  I'd be happy to help, but since I do not have a Teensy 3.5 or 3.6, my local vendor (Mandu) does not have 3.6's in stock, and none of my friends have a 3.6 they can lend me, I can only write the example code for Teensy 3.2; the same may or may not work for Teensy 3.5 or 3.6. (Looking at the above source projects, I believe it should because Teensyduino abstractions would be used, but I'd be no help in debugging it on Teensy 3.5 or 3.6 if any issues do occur.)

[Nominal Animal]

Indeed, I tested DMA-based ADC on Teensy 3.2.  (Didn't bother to wire the conversion trigger yet, though, only the DMA part.)

The only really annoying thing was to map the pins to the ADC channels: you start from the pin name/number on the board pinout, then look up its datasheet pin name from the simplified schematics.  Then, you look up the datasheet pin name in the Signal multiplexing and Pin Assignment chapter in the relevant manual, and see if the default mapping has an analog input signal name beginning with ADC0 or ADC1.  If it does, that ADC can sample that input pin; just look up the analog input signal name in the datasheet ADC0 and ADC1 channel assignment sections (in the same manual), but note that the channel number is given in binary.  Also, note that Teensy 3.x and LC all have an additional ADC MUXSEL bit that cannot be modified during DMA.  Input signal names ending with 'a' are MUXSEL=0, input signal names ending with 'b' are MUXSEL=1, but all others are the same for both MUXSEL values.

I did the above for only Teensy 3.2:

• ADC0 can sample pins 14=A0, 15=A1, 18=A4, 19=A5, 20=A6, 21=A7, 22=A8, 23=A9, A10, A11, and A14.
• ADC1 can sample pins 27=A16, 28=A17, 29=A18, and 30=A19.
• Both can sample pins 16=A2, 17=A3, A12, and A13
• Both can also sample the internal temperature sensor, and the internal 1.2V reference voltage.
• To get the same sample rate for all pins, both should sample the same number of pins (can "pad" by sampling the reference voltage)
• Using DMA, neither can sample pin 26=A15 or pin 31=A20, since they use a different MUX selector ('a', the others are all 'b' or MUXSEL=1).

For 16-bit samples, the ADC clock must not exceed 12 MHz.  It is normally derived from the CPU clock by dividing by 2ⁿ, with certain limits on n.  For maximum sample rate for 16-bit samples, you'll want to run Teensy 3.2 at 48 MHz, Teensy 3.5 at 96 MHz, and Teensy 3.6 at 96 MHz (or 6.7% overclocked at 192 MHz).

For 12-bit samples, the ADC clock must not exceed 18 MHz.  For maximum sample rate, you'll want to run Teensy 3.2 and Teensy 3.5 at 72 MHz, and Teensy 3.6 at 144 MHz.

There is a separate 16 MHz clock not dependent on the CPU clock, which can be used for the ADC if using 12-bit samples.  Then, you can run Teensy 3.5 at 120 MHz, 96 MHz, or 72 MHz; Teensy 3.6 at 180/168/144 MHz.

If the analog voltage is from an instrumentation amplifier, the sampling time does not need to be stretched.  If the total sample rate allows, the ADC units can automatically average 4, 8, 16, or 32 samples before storing, to reduce noise (for n = 2, 3, 4 and 5, respectively; n = 0 without averaging).

A single circular buffer per ADC suffices, as there is a hardware register that tells where the DMA is filling the data.  To avoid interfering with normal operations, the TPM2 timer/pwm module looks like a good choice.

By default, at high sample rates, the analog signal is sampled 2+6×2ⁿ ADC clock cycles, and converted in 5+22×2ⁿ (12-bit) or 5+27×2ⁿ (16-bit) ADC clock cycles and 5 bus clock cycles.  DMA takes probably a couple of bus clock cycles too.

So, for 12-bit samples, one can estimate 17+28×2ⁿ ADC clock cycles, and for 16-bit samples, 17+33×2ⁿ ADC clock cycles, including the bus cycle estimates and the DMA overhead. Without averaging this means 400,000 total 12-bit samples per second per ADC, or 240,000 total 16-bit samples per second per ADC.

If one were to oversample (at, say, 40,000 samples/second per input), it would be rather trivial to average them when saving or passing over USB.

The sampling pattern can be freely chosen, too; it does not need to do round-robin over the chosen analog input pins.  This means that using a pattern like 'ABACABAC' A has 3x the sample rate of B or C, half the total sample rate.

[powerfly]

So after much deliberation, I think I'm going to go with the openlogger ( https://store.digilentinc.com/openlogger-high-resolution-portable-data-logger/ ). Thanks Rstoffer for positing the link to it eariler in this thread

I'm up against time here and the project (it's part of my education) isn't about coding/electronics, this is supposed to be a device to use in experiments to record data, it's the data that I'm supposed to write about in the project, not how the thing that records it works. I've had a look at trying to code the teensy to use it as a data logger but so far I haven't got it to work at the required rate, and also the openlogger seems to be able to process the data in a more readable way which likely will save a lot of time in the post processing stage, and as previously mentioned time is a priority here.

The openlogger seems to be able to do what I need, 8 channels, all can go over 10kHz and it can read the data straight to an SD card. the only issue is converting the dlog file to a .csv file or something, which some users of this device have complained is complicated (look at the ratings). If any of you have experience with using this device, or converting dlog files, that would be helpful. I'm really hoping this device will work as doing it with a teensy seems daunting for someone with basically no coding experience or experience using microcontrollers.

[Nominal Animal]

I'm up against time here and the project (it's part of my education) isn't about coding/electronics, this is supposed to be a device to use in experiments to record data, it's the data that I'm supposed to write about in the project, not how the thing that records it works.

You really should have mentioned this in your initial message!  OpenLogger definitely looks like a good tool for your use case, in my opinion.

I've had a look at trying to code the teensy to use it as a data logger but so far I haven't got it to work at the required rate, and also the openlogger seems to be able to process the data in a more readable way which likely will save a lot of time in the post processing stage, and as previously mentioned time is a priority here.

It isn't trivial, and selecting the optimum settings (sampling duration, averaging) depends a lot on the input circuits.  It is definitely a project on its own.

The openlogger seems to be able to do what I need, 8 channels, all can go over 10kHz and it can read the data straight to an SD card. the only issue is converting the dlog file to a .csv file or something, which some users of this device have complained is complicated (look at the ratings).

It looks like you might need to download and compile the dlog-utils yourself, as the Agent software doesn't have it implemented yet (as of April 2019).

If you have Python 3, reading the binary format and emitting whatever you want is very straightforward and simple: you open the binary file in binary format (i.e., you read bytes, not strs).  (The same code will work on all OSes.) The first 512 bytes of the file contain the header.  You can convert these to integer values using unpack from the built-in struct library.

OpenScope uses 16 bits per sample, cChannels channels, sampled at xSPS/sampleFreqUnits samples per second, actualCount samples (per channel) in the file.  Each sample is a linear function of sampled voltage, with 1 V = voltageUnits (if you divide each sample by voltageUnits, you get the sampled voltage.)

Likely the most common format you could use is the pure text tabulated format, with one set of samples per line.  First column is typically time (in seconds relative to the start of the measurement is common), followed by the actual physical quantities measured via the voltage levels, optionally followed by the actual voltage levels.  This way, when you attach the dataset to your report, anyone can read it into whatever application they want, from Matlab/Octave to Excel/Calc to Gnuplot/xfig/etc. to custom programs.  (CSV is not as common for purely numeric data.)