PlainDAQ - open source DAQ module for Raspberry Pi Pico

[palpurul]

Hello,

I've pre-launched another project on crowdsupply. It's a simple open-source DAQ module for the Raspberry Pico.

Subscibe to PlainDAQ, if you want to get notified when we go live: https://www.crowdsupply.com/kuncu-teknoloji/plaindaq

It's a simple board with 4 analog inputs and a single analog output, hence the PlainDAQ  , I still doubt if it's a good name for a product.  . It can also output +/- supply voltage for powering circuits that need symmetrical power supply. Optionally it comes with ESP-WROOM-02 (wi-fi) module.


Here are the updated pictures! 👇👇👇






Brief:


Components:


Now Let's talk about the precision stuff 
Note: I will update this section as I verify any specifications that is specified right now.

The PCB is on its way, right now it's stuck in the customs for some reason 

Analog Inputs:

• 12-bit resolution
  
• Max. 500 ksps samling rate
• SNR, ENOB, noise: To be determined
• Differential Input ADC (Increase noise immunity)[/i]
• 4 high-impedance buffered channels (low bias current)
• 4 bipolar ranges. ±0.5 V, ±1 V, ±2 V, ±4 V
• Auto-offset calibration
• Bandwidth: To be determined

Voltage Reference:

• 2.5 V, 20 ppm/°C (Typical) Drift
• Calibrate to 0.05 %, Stored in as ROM as calibration data

Analog Output:

• Single channel
• Buffered low-impedance output
• 10-bit resolution
• SNR, ENOB, noise: To be determined
• Max. 50 ksps samling rate
• Bandwidth: To be determined

Bipolar Voltage Supply:

• +/-5V supply voltage
• +/-100mA current rating

I am trying to make it as cheap as possible while trying to provide a semi-precision DAQ unit that can be used with Raspberry Pi Pico, so I need to make some compromises between cost and accuracy, it's not very easy 

PlainDAQ will communicate through USB to graph captured waveforms, I will prepare a simple GUI for that purpose. There are lots of things to make in the software side of things (both GUI and embedded). I’d like to list the thing that are not implemented, but will implemented in the future, I share progress log I there is a progress 😊

Things to implement/test:

• USB speed test, gotta reach to at least 6Mbit/s (USB full speed runs 12Mhz ☹) Edit: 22.02.2022 7Mbit/s transder speed is achieved
• Rising/Falling edge trigger
• Graphing software
• Simple Python scripting for controlling PlainDAQ
• Measuring analog inputs’ SNR
• Measuring analog output’s SNR
• Measuring analog inputs’ bandwidth
• Measuring analog output’s bandwidth
• Measure the effect of ESP32's operation on SNR (Added on 14.02.2022, kudos to @Kean)

I will add to the list when I need to add a new feature based on feedback.

Tell me what you think. This is the first Crowdsupply project that I am running, wish me luck 😊.

PlainDAQ also comes with its GUI, here is a sneak-peek of the GUI:

[mikerj]

Is there any kind of protection on the ADC and DAC pins to prevent damage from ESD and overvoltage?  This seems to be a common failing with basic DAQ designs, but is pretty essential for any real world use.

[palpurul]

Is there any kind of protection on the ADC and DAC pins to prevent damage from ESD and overvoltage?  This seems to be a common failing with basic DAQ designs, but is pretty essential for any real world use.

This is the first prototype, honestly at this stage I was rushing to have my hands on something working, so omitted input protection on the board shown in the pictures.

I will keep your suggestion in mind and next revision will include input protections.

Thank you for your suggestion. These kind of feedbacks really help. 

[Kean]

This looks interesting.  Will you be publishing the schematic?  That will help some users like me to understand suitability for an application.

I'd be interested in whether you see any increased noise (lower ENOB) when used with the optional ESP32 & WiFi, assuming the ESP32 is meant to mount on the back of the PCB.  Especially with the 14-bit ADC option.

Just a suggestion - You may want to spend a little extra time on assembly for the next sample PCB.  The soldering is a bit rough on the above pictures, and some of the terminal pins aren't even soldered.

[palpurul]

This looks interesting.  Will you be publishing the schematic?  That will help some users like me to understand suitability for an application.

It's all open-source. I will release the whole KiCAD project. I will also release all of the firmware that I wrote for it.

I'd be interested in whether you see any increased noise (lower ENOB) when used with the optional ESP32 & WiFi, assuming the ESP32 is meant to mount on the back of the PCB.  Especially with the 14-bit ADC option.

That's an interesting point. I've never thought about it actually, it would probably increase the noise. Obviously this increase is going to be more pronounced at higher resolution as you pointed out. ESP32 module is something that I added just before the pre-launch page is released, actually the boards that you see in my post does not include the ESP32 module yet, It's for another update. The board is a 4-layer board and the ESP32 module will be on the bottom side, so maybe ground plane would help to combat this issue.
Anyways, it's going to be fun to test it and I will definitely post the results here  (added to Things to implement/test list)

Just a suggestion - You may want to spend a little extra time on assembly for the next sample PCB.  The soldering is a bit rough on the above pictures, and some of the terminal pins aren't even soldered.

I know a person with a keen eye would someday notice the flaws in my hand assembly, you are first, so congrats 
I didn't solder all of the pins because I had a single sample at the time I was taking the pictures and I needed it take its pictures of it with and without the raspberry pi pico on top, and therefore I left most of the pins unsoldered.
Excuses excuses I know. 

Consider subscribing to mailing. I will post any updates here, but if you want to get notified you can sub here

[Kean]

It's all open-source. I will release the whole KiCAD project. I will also release all of the firmware that I wrote for it.

Great!

That's an interesting point. I've never thought about it actually, it would probably increase the noise. Obviously this increase is going to be more pronounced at higher resolution as you pointed out. ESP32 module is something that I added just before the pre-launch page is released, actually the boards that you see in my post does not include the ESP32 module yet, It's for another update. The board is a 4-layer board and the ESP32 module will be on the bottom side, so maybe ground plane would help to combat this issue.

Yes, it looked to me that the current footprint on the back looks like one for a BLE module not ESP32-MINI.  My main concern with the ESP32 would be current paths as the ESP32 chip can draw quite a bit when transmitting.  Depending on the application it may be possible to avoid ADC conversions during application based transmission, but not all due to the nature of TCP/IP & WiFi.

I didn't solder all of the pins because I had a single sample at the time I was taking the pictures and I needed it take its pictures of it with and without the raspberry pi pico on top, and therefore I left most of the pins unsoldered.
Excuses excuses I know. 

Reasonable excuses.  No point making too many prototypes when you are planning another revision soon.

Consider subscribing to mailing. I will post any updates here, but if you want to get notified you can sub here

Haha, already done on both projects - but this one is of more interest to me right now.  Always plenty of applications for DAQ here.

[palpurul]

Yes, it looked to me that the current footprint on the back looks like one for a BLE module not ESP32-MINI

Good catch! It was BLE module from RAYTAC. It is almost unobtanium right now, so switched to ESP32s.

Reasonable excuses.  No point making too many prototypes when you are planning another revision soon.

Thanks!

My main concern with the ESP32 would be current paths as the ESP32 chip can draw quite a bit when transmitting.  Depending on the application it may be possible to avoid ADC conversions during application based transmission, but not all due to the nature of TCP/IP & WiFi.

Yes they are quite power hungry, can go up to 320mA as datasheet specifies. It's probably not constant current draw, most certainly large spikes of currents which makes it even more important in terms of conserving SNR.

Haha, already done on both projects - but this one is of more interest to me right now.  Always plenty of applications for DAQ here.

Means a lot to me, thanks! I keep posting update once I made some interesting progress! 

[palpurul]

Progress!
PlainDAQ needs transfer lots of data and it's gotta do it fast.

I wanted to test how fast I can tranfer data via USB. I used two raspberry pi pico's one of them doing the debugging and the other is doing the actual work.

Ultimately I am going to need to transfer 6MBit/s of data from pico to computer (12-bit 500ksps ADC = 6Mbits/s)

The Test Setup

Pico on the lower side of the picture is debugging and the other is the DUT (device under test)

USB Speed Test

The USB is in bulk transfer mode. I don't really know much about inner workings of the USB protocol, but it allows me to send 64 bytes at a time, and therefore I needed be clever about transferring 1000s of 64 bytes packages.

I imported the the example code provided in raspberry pi's github page and modified it in a way that I can receive multiple packages easily. To do that I erased every print statement (they are SLOW!) and implemented a counter to keep track of number of packages sent.

You can check my code here. I also included the python file to work with it as well.

And finally, here are the results:



I can reach up to more than 7Mbits/s. The speed depends on your setup as well. For example, if you use a hub it's slower.

I can't really complain about this. I don't think I can go faster than this because at the end of the day USB full-speed works at 12 MHz.

Did anyone managed get more than 7Mbits/s? I am curious.

That's it for this update!

[Kean]

To get higher throughput you could potentially use some simple compression.  A lot of ADC data at high data rates may just be varying by small amounts due to noise, unless of course you are actually measuring a signal that is some decent fraction the bandwidth limit.

If you are packing 12-bit data already, you could use some other scheme to pack small changes in smaller number of bits.  I'm sure there is a variation on RLE or similar which can compress the noise in a lossless way.  It is a trade off between processing speed vs communications speed vs use case.

[SiliconWizard]

I can reach up to more than 7Mbits/s. The speed depends on your setup as well. For example, if you use a hub it's slower.

I can't really complain about this. I don't think I can go faster than this because at the end of the day USB full-speed works at 12 MHz.

Did anyone managed get more than 7Mbits/s? I am curious.

I'm curious as well. What kind of driver are you using on the PC side?

[palpurul]

To get higher throughput you could potentially use some simple compression.  A lot of ADC data at high data rates may just be varying by small amounts due to noise, unless of course you are actually measuring a signal that is some decent fraction the bandwidth limit.

I've never thought about using compression because I know nothing about it. If you could tell me where I should start, I would be more than happy 

If you are packing 12-bit data already, you could use some other scheme to pack small changes in smaller number of bits.  I'm sure there is a variation on RLE or similar which can compress the noise in a lossless way.  It is a trade off between processing speed vs communications speed vs use case.

I'm not sure what RLE is 

[palpurul]

I'm curious as well. What kind of driver are you using on the PC side?

I'm not really sure, I used python USB module maybe that means that I am using whatever the default driver is... I don't have the board with mw right now once I got my hands on it, I am going to look into it.

[Kean]

Yeah, I'm not an expert on compression.  It can get into heavy mathematics beyond my pay grade quite quickly.

RLE is a simple compression scheme which can work well with data that has a lot of repeated values in a series, such that it stores the values and repeat counts.  It is /was commonly used in simple image formats where a lot of a single background color may be present.  Fax machines used something similar.  It is easy to implement and pretty fast, but when used with certain data it could increase the overall size in which case you just send the original uncompressed data.  See https://en.wikipedia.org/wiki/Run-length_encoding

Maybe Deflate compression as used in the original PKZIP (a bit more complex).  A lossless audio codec may be quite suitable as they are also waveforms.

There are plenty of options without re-inventing the wheel, but I don't know enough to make a recommendation.
See https://en.wikipedia.org/wiki/Lossless_compression

[palpurul]

Yeah, I'm not an expert on compression.  It can get into heavy mathematics beyond my pay grade quite quickly.

RLE is a simple compression scheme which can work well with data that has a lot of repeated values in a series, such that it stores the values and repeat counts.  It is /was commonly used in simple image formats where a lot of a single background color may be present.  Fax machines used something similar.  It is easy to implement and pretty fast, but when used with certain data it could increase the overall size in which case you just send the original uncompressed data.  See https://en.wikipedia.org/wiki/Run-length_encoding

Maybe Deflate compression as used in the original PKZIP (a bit more complex).  A lossless audio codec may be quite suitable as they are also waveforms.

There are plenty of options without re-inventing the wheel, but I don't know enough to make a recommendation.
See https://en.wikipedia.org/wiki/Lossless_compression

I will look into into, that going to be really usefull for me becuase I am plaining to transder 14-bit 500 ksps (7Mbits) data through USB full-speed for another project.
The most important thing here is how easy they are to implement in a microcontroller. I'll definitely look into it. As you mentioned I'm sure that there are many implementations I can find on github

[palpurul]

UPDATE, FIRST STEP IN DEVELOPING GUI.

I decided to start developing the GUI because it made sense to develop the GUI to some extend to start measuring SNR, Bandwdith, accuracy etc, but before that I needed to know how matplotlib is going to perform under stress and I wrote a simple script to run stress test.

Let me know if you have any suggestions about developing the GUI, do you think 20 FPS is enough for such application? What other easy-to-use options to use for developing GUI?

The script generates a 2000 sample long sinewave + noise and tries to update the screen every millisecond. It can achieve 60 FPS, although is not very consistent, but I think this is enough for developing a GUI for plainDAQ.

If you need performance using matplotlib use animation and blitting, they make things considerably faster





I know that matplotlib is not meant to be used for such applications (real-time graphing), but it's easy to use and it runs quite smoothly at 20 FPS which I think is enough.

Here is the code. I am an hardware guy mostly, python is very new to me. I used examples provided by matplotlib mostly. If you're going to criticize my code, go easy on me 
Code: stress test code

Subscibe to PlainDAQ, if you want to get notified when we go live.

[Berni]

DAQ boards are usually used as part of some automated test setup. So being able to display the waveform like an oscilloscope is not that useful in most cases. Tho it is still an excellent demo program to show it working and quickly test it out.

What tends to be more important is being able to easily interface to it in the programing language of choice. So you want to use a driver library that has some multiplatform support, so that people using say C++ or C# can use it.

When it comes to matplotlib it is pretty performant since a lot of these chart libraries do the actual piloting in machine code DLLs rather than python. When it comes to hitches in performance this might actually the the fault of pythons garbage collector. Once you throw away enough allocated memory the garbage collector kicks in to reclaim it as free memory. If you built up a lot of garbage it can take a significant amount of time to clean up and during this time the program is paused. If that is a problem there are ways to manually run garbage collection before it gets too bad, or even better to write code in a way that generates less garbage.

[palpurul]

DAQ boards are usually used as part of some automated test setup. So being able to display the waveform like an oscilloscope is not that useful in most cases. Tho it is still an excellent demo program to show it working and quickly test it out.

Thanks for your feedback! I understand your point, but hear me out  I am also trying to build something really affordable for people to use as a basic oscilloscope/waveform gen/voltmeter maybe I can add some digital I/O for triggering etc. What I am trying to say is I am not trying build only a daq to be used in automated testing, I also want to offer a semi-precision, basic, all around tool and to support this I think a GUI can help to demonstrate these features. GUI can also help people to play with it as soon as they get their hands on PlainDAQ 

What tends to be more important is being able to easily interface to it in the programing language of choice. So you want to use a driver library that has some multiplatform support, so that people using say C++ or C# can use it.

I'll keep that in mind, that's an important suggestion for me. Building something multi platform would be quite difficult for me, I am writing almost everything in python at the software side and seems to be the most popular language in the maker community. Do you have a favorite DAQ module that comes with a good driver/API? It can be really helpful for me to get an idea about what people are used to in the industry.

When it comes to matplotlib it is pretty performant since a lot of these chart libraries do the actual piloting in machine code DLLs rather than python. When it comes to hitches in performance this might actually the the fault of pythons garbage collector. Once you throw away enough allocated memory the garbage collector kicks in to reclaim it as free memory. If you built up a lot of garbage it can take a significant amount of time to clean up and during this time the program is paused. If that is a problem there are ways to manually run garbage collection before it gets too bad, or even better to write code in a way that generates less garbage.

I am learning a lot from you! As I said before I am not too much of a software guy, but I am kind of eager to learn new territories. I have had no experience with garbage collection. I'll definitely look into learning basics of garbage collection in python or in general. At the end of the day there is a possibility for it to run faster after all?

[Berni]

The industry standard for test equipment automation is mostly VISA:
https://en.wikipedia.org/wiki/Virtual_instrument_software_architecture
Even more compatibility can be achieved with using SCPI on top of it:
https://en.wikipedia.org/wiki/Standard_Commands_for_Programmable_Instruments

Thanks to these the same piece of PC test automation software can use a Keysight DMM, Fluke DMM or Keithley DMM without being specifically designed for any of them. The communication can even happen over GPIB, LAN, USB, RS232 ..etc and the automation software doesn't even know about it. The software simply asks to open connection to a device named for example "GPIB0::5::INSTR" and the VISA API will figure out where that piece of test equipment is and how to communicate with it. Once connected the connection looks pretty much like a serial port that can send and receive any data you want.

The neat thing is that since RS232 is one of the commonly supported VISA communication methods, means that any serial port can be set up to be a VISA device. This includes USB to UART chips that use the generic USB CDC driver. So if you also speak the SCPI protocol over that serial port your DAQ can be used with existing software that already uses DAQs from the big vendors like Keysight,Tek...etc

The way to use VISA is to just install your favorite implementation of VISA from a vendor like NI, Keysight, Rigol etc.. My personal favorite is the Keysight VISA implementation since it has a nice UI and some useful debugging tools to spy on the communication happening between devices. This does not lock you into only using Keysight gear, it talks to a Rigol DMM just fine (It just makes setting up some of the more complex GPIB or PXI stuff easier on Keysight branded equipment)

This compatibility goes both ways, so if your Python GUI uses VISA to talk to the instrument means that it can talk to other DAQ boards too, not just yours.

As for pythons garbage collection, very few people give it much attention since among the popular languages python is the slowest of them. It is a language that focuses on being easy and powerful rather than being performant. Java is faster, C# is even faster while both being garbage collected. Lower level languages like C and C++ are even faster and don't need garbage collection, but they are harder to use. My personal preference is C# because it comes with lots of modern language creature comforts, is still rather performant and comes with VisualStudio (Excellent free official IDE that makes GUI development super easy and has some of the best code autocomplete)

[palpurul]

The industry standard for test equipment automation is mostly VISA:
https://en.wikipedia.org/wiki/Virtual_instrument_software_architecture
Even more compatibility can be achieved with using SCPI on top of it:
https://en.wikipedia.org/wiki/Standard_Commands_for_Programmable_Instruments

Thanks to these the same piece of PC test automation software can use a Keysight DMM, Fluke DMM or Keithley DMM without being specifically designed for any of them. The communication can even happen over GPIB, LAN, USB, RS232 ..etc and the automation software doesn't even know about it. The software simply asks to open connection to a device named for example "GPIB0::5::INSTR" and the VISA API will figure out where that piece of test equipment is and how to communicate with it. Once connected the connection looks pretty much like a serial port that can send and receive any data you want.

The neat thing is that since RS232 is one of the commonly supported VISA communication methods, means that any serial port can be set up to be a VISA device. This includes USB to UART chips that use the generic USB CDC driver. So if you also speak the SCPI protocol over that serial port your DAQ can be used with existing software that already uses DAQs from the big vendors like Keysight,Tek...etc

The thing that I am wondering is that how they manage to transfer large measurement data to computer over serial terminal emulation (USB UART chips are not very fast). I also have a spectrum analyzer with USB 3.0 connection (RSA503A) I think it supports visa and SCPI and obviously it can manage very fast data transfer to PC. I thought all SCPI protocols are implement in serial terminal emulation. 

The way to use VISA is to just install your favorite implementation of VISA from a vendor like NI, Keysight, Rigol etc.. My personal favorite is the Keysight VISA implementation since it has a nice UI and some useful debugging tools to spy on the communication happening between devices. This does not lock you into only using Keysight gear, it talks to a Rigol DMM just fine (It just makes setting up some of the more complex GPIB or PXI stuff easier on Keysight branded equipment)

This compatibility goes both ways, so if your Python GUI uses VISA to talk to the instrument means that it can talk to other DAQ boards too, not just yours.

I think I've used SCPI or visa with my DMM7510 in the past also in python. I used it in digitizer mode and acquired large data set for harmonic distortion evaluation. Maybe I can work on getting plainDAQ to work with it. It could be interesting, but again I am kind of aiming the hobby market as PlainDAQ seems more suitable for that spesific audience and I am not sure if I should prioritize getting PlainDAQ to be compatible with VISA or SCPI. Still it would be a serious addition!

As for pythons garbage collection, very few people give it much attention since among the popular languages python is the slowest of them. It is a language that focuses on being easy and powerful rather than being performant. Java is faster, C# is even faster while both being garbage collected. Lower level languages like C and C++ are even faster and don't need garbage collection, but they are harder to use. My personal preference is C# because it comes with lots of modern language creature comforts, is still rather performant and comes with VisualStudio (Excellent free official IDE that makes GUI development super easy and has some of the best code autocomplete)

Yes! I watched lots basic python tutorials, but none of them mentioned garbage collection. Even the term is very foreign to me right now. 

[Berni]

Well using SCPI is more about using an existing well supported protocol as opposed to creating a new proprietary protocol (that nobody is familiar with and requires specific software). Someone who is familiar with VISA will be right at home (quite a few people are since modern SCPI capable test gear is a lot more accessible cheep from ebay these days), if not they have to figure it out. With a proprietary protocol they always have to figure out how to use it.

Just because something is serial does not mean it is slow. For example TCP/IP connections look much like two machines connected by a serial port. Yet network file sharing will happily transfer at a speed of 1000 GB/s from your NAS server (provided you have a pretty beastly NAS and a 10Gbit network card). When using VISA over LAN it is also just a TCP/IP connection that will also transfer >1000 GB/s if the instrument you are connecting to is fast enough (Tho i never seen a 10Gbit network card on test equipment so far).

The thing that makes physical serial ports slow is the low baudrate they run at. However USB 2.0 HS always runs at 480Mbit. This includes USB to Serial dongles. The data gets sent to the dongle in 480Mbit bursts that get FIFO buffered by the dongle and sent out as UART at the desired baud rate. If the UART can't keep up then the FIFO fills up and the dongle tells the PC to wait. So this means there is no actual speed limit for a USB CDC virtual serial port. Even if the serial port is configured for 9600 baud this just means the PC driver sends a "Please configure your self for 9600 baud 1 stop bit, no parity" packet, but the PC will feed the device as much data as it will accept. So if the FIFO is being consumed by something faster than a 9600 baud UART the PC will keep sending, even if this is 200Mbit/s. Introducing yourself as a "USB Serial device" simply tells Windows that it should assign a COM port to you so that windows applications can use the serial port API to talk to it.

The actual speed limit is the driver latency, since USB 2.0 controllers are annoying for operating systems to deal with. They don't have very large buffers and need a fair bit of babysitting from the CPU, making it annoying for multitasking OSes like Windows to service fast enough to prevent pauses in the data transfer. This is one of the things that USB 3.0 fixes, allowing for most of those 5/10Gbit of bandwidth to be efficiently used.

Tho in your case the Pi Pico only has USB 1.1 so you are limited to the 12Mbit FullSpeed USB, but you should be able to get pretty close to the theoretical 12Mbit if you use it in an efficient way (large data blocks, minimized latency etc..)

[palpurul]

Tho in your case the Pi Pico only has USB 1.1 so you are limited to the 12Mbit FullSpeed USB, but you should be able to get pretty close to the theoretical 12Mbit if you use it in an efficient way (large data blocks, minimized latency etc..)

I'm getting 7Mbits/s with this code at most (you can check the earlier replies to see the results). What do you think I should do to make it faster? I am using the largest package size allowable (64 bytes) in bulk transfer.

[Berni]

Here are some benchmarks for USB speed https://www.pjrc.com/teensy/usb_serial.html

Turns out there is a rule on USB that should only utilize 90% or 80% of the bus bandwidth (depending on what spec you read since USB 2.0 updates FullSpeed operation a bit) and there is a bit of overhead on top of that. So about 10Mbit is the theoretical maximum and about 9.2Mbit is what he could actually get. So there is room for improvement. It is possible that the driver for the USB device peripheral on your pico is not the most optimized (squeezing the most out of hardware can sometimes take some trickery).

Interestingly OSes are still finding it hard to service USB controllers, so this drops the usable bandwidth some too. Likely because USB FS does not support the bigger 512byte long packets that USB HS has. So the CPU has to do too much babysitting to get full bandwidth.

Oh and as usual USB hubs are always to be weary of. They are known to cause problems in high bandwidth situations. Some motherboards even have on board USB hub chips in order to offer more USB ports.

EDIT:
Also it might not be a good idea to push USB to the limits in a product. I have used the Saleae USB logic analyzers quite a bit. Those stream data live to the PC and use the PCs RAM as the sample memory. When using the max sample rate these, they get unreliable. In favorable conditions (low CPU load, good USB controller, no hubs etc...) it works well, but otherwise it keeps clogging the FIFO, making that sample rate unusable.

[palpurul]

Here are some benchmarks for USB speed https://www.pjrc.com/teensy/usb_serial.html

Turns out there is a rule on USB that should only utilize 90% or 80% of the bus bandwidth (depending on what spec you read since USB 2.0 updates FullSpeed operation a bit) and there is a bit of overhead on top of that. So about 10Mbit is the theoretical maximum and about 9.2Mbit is what he could actually get. So there is room for improvement. It is possible that the driver for the USB device peripheral on your pico is not the most optimized (squeezing the most out of hardware can sometimes take some trickery).

Interestingly OSes are still finding it hard to service USB controllers, so this drops the usable bandwidth some too. Likely because USB FS does not support the bigger 512byte long packets that USB HS has. So the CPU has to do too much babysitting to get full bandwidth.

I think at that point I am in the limits of the spesific implementation of the USB hardware in the MCU, PC side and the USB driver itself. In the example code that I posted I send the largest package with almost doing nothing between each package. Maybe I am missing something, but I assume this is the limit with my configuration.

Oh and as usual USB hubs are always to be weary of. They are known to cause problems in high bandwidth situations. Some motherboards even have on board USB hub chips in order to offer more USB ports.

Thanks for letting me know this. I observed that when I connected PlainDAQ to hub, it gets slower. I can only achieve 7Mbit/s when I connect right at my laptops USB port.

EDIT:
Also it might not be a good idea to push USB to the limits in a product. I have used the Saleae USB logic analyzers quite a bit. Those stream data live to the PC and use the PCs RAM as the sample memory. When using the max sample rate these, they get unreliable. In favorable conditions (low CPU load, good USB controller, no hubs etc...) it works well, but otherwise it keeps clogging the FIFO, making that sample rate unusable.

I'll keep that in mind. Unfourtanetly I need close to the max bit rate I'm getting, I need 6MBit/s and I can at most get 7MBit/s. Earlier @Kean suggested me to use compression. Maybe I can get away with 5Mbit/s with compression, but I'm too sure which one to use or more importantly how to implement them.

[Berni]

Yeah having USB HS 480Mbit here would be very useful, but MCUs with a built in USB HS PHY are quite rare.

Compression for noisy waveforms is a bit tricky since it works terribly with the simple and easy RLE compression. From my experience wavelet transforms work well for compressing continuous signals. This in itself doesn't make the result smaller, but it does make a lot of values near 0. Allowing you to round them to 0 and then have RLE work very well on it(This is mostly how JPEG works). This is obviously lossy so to make it lossless you also need to include the error signal (that is much smaller so needs fewer bits).

Tho more of a problem is that lossless compression by definition can't grantee a given compression ratio. If you feed it a messy enough uncompressible signal then you simply get no compression or even negative compression. In your case this would make the FIFO overflow and everything to break down.

But if you use lossy compression you can give it a target bitrate, so the data is squeezed down however much is needed to fit. If you give it a particularly uncompressible signal then the result is a particularly distorted signal that still fits in the bitrate. For a wavelet transform this is easily done by adjusting at what threshold data points are rounded to 0 in order to make RLE more efficient. You can often squeeze down the amount of data to only 10% its original size with this.

More of a problem is that lossy compression is generally not desired for a DAQ. In 99% of use cases the lossy compression would not be noticeable at all, especially since it would be very good at compressing the kind of signals it usually gets fed. If anything the data might look better in most cases since the tiny high frequency noise is the first thing the compression is tempted to throw away. But i would imagine quite a few people would hate knowing in the back of there mind that there precious data might have been mucked up in compression along the way.

[palpurul]

Yeah having USB HS 480Mbit here would be very useful, but MCUs with a built in USB HS PHY are quite rare.

I know how rare a MCU with USB HS. Controller chips are also not that common and expensive...

Compression for noisy waveforms is a bit tricky since it works terribly with the simple and easy RLE compression. From my experience wavelet transforms work well for compressing continuous signals. This in itself doesn't make the result smaller, but it does make a lot of values near 0. Allowing you to round them to 0 and then have RLE work very well on it(This is mostly how JPEG works). This is obviously lossy so to make it lossless you also need to include the error signal (that is much smaller so needs fewer bits).

Tho more of a problem is that lossless compression by definition can't grantee a given compression ratio. If you feed it a messy enough uncompressible signal then you simply get no compression or even negative compression. In your case this would make the FIFO overflow and everything to break down.

But if you use lossy compression you can give it a target bitrate, so the data is squeezed down however much is needed to fit. If you give it a particularly uncompressible signal then the result is a particularly distorted signal that still fits in the bitrate. For a wavelet transform this is easily done by adjusting at what threshold data points are rounded to 0 in order to make RLE more efficient. You can often squeeze down the amount of data to only 10% its original size with this.

More of a problem is that lossy compression is generally not desired for a DAQ. In 99% of use cases the lossy compression would not be noticeable at all, especially since it would be very good at compressing the kind of signals it usually gets fed. If anything the data might look better in most cases since the tiny high frequency noise is the first thing the compression is tempted to throw away.

This went over my head quite quickly. 
For now I just skip this concept all together because it's a very foreign concept for me.
I must thank you for your contribution, I've learnt a lot from you.

i would imagine quite a few people would hate knowing in the back of there mind that there precious data might have been mucked up in compression along the way.

I can relate