Simplest way for 75MS/s 14bit recordings to SSD?

[240RS]

I want to sample baseband RF at 75MHz and record 14bit samples (real values only) on SSD/HD for later analysis. Ideally for a minute or so.

I was hopeful I could do this with a red Pitaya directly to its SD card, but that seems limited to 10MB/s. I assume I would need 150MB/s (2 bytes per sample).

I had overlooked this issue. I was simply thinking ADC’s are fast enough these days, USB3.0 is fast enough. But I can literally not find a single solution.

Is there something obvious for the recording part?

Later on. I would like to be able to play it back. I just purchased an eBay Anritsu MG3700a vector signal generator that can play back 512MS with max 160MHz bandwidth. But maybe there is a logical DAC solution also.

Thanks

[radiolistener]

I tried to get 30 MS/s 14 bits (aligned to 16 bits) through gigabit ethernet on i5 760 running at 3200 MHz. But it works unstable, I have packet drops. I think there is needs some ethernet card with hardware acceleration, because my onboard Realtek 8112L gigabit chip is unable to handle continuous realtime streams above 700 MBit/s on Windows 7 without high CPU load even when receiver app is not running. I just start OS and don't run anything, then push 700-900 MBit/s stream through ethernet cable and my PC starts to lagging despite the fact that it has pretty fast 95 Watt desktop CPU with massive heatsink.

USB2 interface just unable to handle more than 480 Mbit (include protocol overhead). I'm not sure if USB3 interface allows to process more fast stream than 1 Gbit in realtime, but anyway 75 MS/s at 14 bits is not so easy for desktop PC, because even fast server HDD is unable to write data continuously at that speed, of course SDD probably can do it, but there is still bottleneck with CPU and motherboard bus speed.

I think there is a sense to implement direct interface between FPGA and SSD to write data directly to SSD with no need to process it on PC.

[langwadt]

https://opalkelly.com/products/xem7310mt/

and a board with your ADC (DAC) ?

[240RS]

I tried to get 30 MS/s 14 bits (aligned to 16 bits) through gigabit ethernet on i5 760 running at 3200 MHz. But it works unstable, I have packet drops. I think there is needs some ethernet card with hardware acceleration, because my onboard Realtek 8112L gigabit chip is unable to handle continuous realtime streams above 700 MBit/s on Windows 7 without high CPU load even when receiver app is not running. I just start OS and don't run anything, then push 700-900 MBit/s stream through ethernet cable and my PC starts to lagging despite the fact that it has pretty fast 95 Watt desktop CPU with massive

What ADC board did you use?

[240RS]

https://opalkelly.com/products/xem7310mt/

and a board with your ADC (DAC) ?

Sounds good indeed, but steep learning curve?

[KE5FX]

https://opalkelly.com/products/xem7310mt/

and a board with your ADC (DAC) ?

Sounds good indeed, but steep learning curve?

Opal Kelly's FrontPanel architecture is very easy to learn and use.  It's proprietary, though, so you have to be prepared to either stick with it in the long run or migrate eventually when your needs change.  (For one thing, your supply chain will be joined at the hip to theirs.)

Also see the ZTex and Trenz boards for a more open, DIY-friendly alternative.  I've used both ZTex and Opal Kelly's products and usually find ZTex a better fit for my needs and interests, but Opal Kelly's products are also superb.

[240RS]

Ah, great. What ADC to connect? That will be parallel data lines I guess?

[langwadt]

Ah, great. What ADC to connect? That will be parallel data lines I guess?

if you have to ask I doubt it is for you ...

how much are you wiling to spend? 

maybe something like this?

https://spectrum-instrumentation.com/products/families/44xx_m4i_pci.php
https://www.alazartech.com/en/product/ats9628/642/

[240RS]

Ah, great. What ADC to connect? That will be parallel data lines I guess?

if you have to ask I doubt it is for you ...

I got that indeed :-). No offense.

Starting point was: 14bit 125MS/s ADC’s are 20-40usd these days. USB3.0 is fast enough. I had just overlooked there isn’t a couple hundred dollar complete solution that would just “dump” 75MS/s on my computer. Had really hoped I would not have to program fpga’s and hookup SYZYGY boards.

[radiolistener]

What ADC board did you use?

I'm using self made board with AD6645-105 ADC. It works good, but has too high power consumption (about 1W from 5V) and pretty hot even with heatsink on it. More modern ADC allows to reduce power consumption.

That will be parallel data lines I guess?

75 MHz x 14 bit will produce at least 1.05 Gbps serial stream, I think it will be hard to deal with it, especially when you prototype it. Even 100 MHz parallel bus has issue if you use a little different wire length.

[ve7xen]

USRP B200 over USB 3 claims 61.55MS/s realtime streaming with 16-bit IQ. Not quite your target but should be very easy to set up, though not as cheap.

Naively, 1Gbps interface of Red Pitaya is not quite fast enough, nor is its SD card interface, so there is not really an 'easy' solution available. If stored as integer, I guess the raw data might be relatively easily compressible, so if you run it through lossless compression and then dump it over the network it can probably keep up, assuming the network interface and CPU can keep up reasonably well with a pretty fat flow, though it's not obvious to me whether or not it would hit the mark. Check the gr-compress module, dump it to a UDP sink and store it on a box with faster disk.

[langwadt]

What ADC board did you use?

I'm using self made board with AD6645-105 ADC. It works good, but has too high power consumption (about 1W from 5V) and pretty hot even with heatsink on it. More modern ADC allows to reduce power consumption.

That will be parallel data lines I guess?

75 MHz x 14 bit will produce at least 1.05 Gbps serial stream, I think it will be hard to deal with it, especially when you prototype it. Even 100 MHz parallel bus has issue if you use a little different wire length.

two pairs of LVDS will do, e.g. https://www.analog.com/en/products/ad9645.html

[larsdenmark]

Perhaps a single board computer with 16 GB of RAM can be used. E.g. the Orange P5 5:
http://www.orangepi.org/html/hardWare/computerAndMicrocontrollers/details/Orange-Pi-5.html

If you can make its GPIO pins work fast enough then you can simply store the data in RAM before moving the data to a SD card or an SSD harddrive.

The Orange Pi 5 has interfaces to network, USB, CD cards and SSDs so there should be plenty of possibilities of getting data out of this thing. Note that it doesn't have Wifi or Bluetooth built in.

[radiolistener]

Perhaps a single board computer with 16 GB of RAM can be used. E.g. the Orange P5 5:
http://www.orangepi.org/html/hardWare/computerAndMicrocontrollers/details/Orange-Pi-5.html

If you can make its GPIO pins work fast enough then you can simply store the data in RAM before moving the data to a SD card or an SSD harddrive.

I think orange pi will be unable to handle even 10-20 MS/s, especially through GPIO, GMII or USB interface. Such a stream needs for at least modern i5/i7 CPU. If you want for a mobile solution, you can try modern notebook for gamers with i7 CPU, but I'm not sure if mobile i7 CPU will be able to handle 75 MS/s because this is more than 1 Gbps realtime continuous stream...

[Nominal Animal]

On non-x86 Linux SBCs:

If you can get a suitable USB 3 DAQ, then Odroid M1 (8 GiB) should work, I believe.  It is a 2.0 GHz quad-core Cortex-A55 with Neon, FPU, and crypto extensions.  There is also an NPU, and Mali G52-2EE graphics (including video acceleration and HDMI to a display).  (I've been considering getting one myself.  I really like Odroid SBCs, and RockChip is one of the rare SoC manufacturers who push support for their SoCs to upstream open source Linux kernels, instead of binding one to vendor kernels only (but HardKernel does provide customized images and kernels too).)

It is based on (specs) Rockchip RK3568, and has a PCIe 3.0 M.2 M-key slot for storage; benchmark (on top of a filesystem in Linux) indicates sequential writes in 512k chunks reaches 1200 MiB/s (so almost an order of magnitude more than you need for 75 Msps @ 16bit = 150 MiB/s).  It also has a 5V SATA (standard SATA connector, but the power connector does not provide 12V), which you can use with SATA SSDs.  SATA is slower than the PCIe, but still has sufficient bandwidth (somewhere between 100 MiB/s and 500 MiB/s, depending on the SATA SSD drive) for your use case.

Instead of USB 3, you could also use MIPI CSI (two differential data input lanes plus clock input to the SoC); see the RK3568B2 Datasheet.  You probably would need a straightforward custom Linux kernel driver to communicate with the MIPI ADC/DAQ, as the standard platform drivers are image-oriented.  Perhaps via Lattice iCE40 FPGA, like using one of the Olimex iCE40 FPGA boards?

I think orange pi will be unable to handle even 10-20 MS/s, especially through GPIO, GMII or USB interface. Such a stream needs for at least modern i5/i7 CPU.

No, Rockchip RK3588S is even more powerful than RK3568.  These puppies have absolutely no problem throwing a gigabyte per second around, continously.
From what end did you pull the belief that an Intel processor would be needed for this?  That's ridiculous, really.

[radiolistener]

No, Rockchip RK3588S is even more powerful than RK3568.  These puppies have absolutely no problem throwing a gigabyte per second around, continously.

I didn't tried that RK3588S, but a much powerful desktop i3 has performance issues even with 20 MS/s. I also tried intel Atom and 2 core ARM Cortex A15, they are just sucks and have problems even with 1 MS/s.

Maybe it can transfer 1-2 Gbps with no issue in some usecases through some internal bus, but if you're planning a real usage through GMII or USB interface, it will starts packet drops at much slower data flow. Just notice that this transfer speed cannot be handled even with gigabit ethernet because it's max transfer speed is smaller than minimum required.

And note that there is required continuous transfer in realtime, not a signle burst transfer. It doesn't allow waiting or delays. Any delay leads to a transfer fail because it leads to a packet drop.  So the real requirement is much higher than 1-2 Gbps in order to cover all other data transfer required for OS functions.

Such a transfer speed requires about 4-8 GB dedicated RAM just for the receive buffer with hardware DMA write.

I didn't tested odroid-m1 or raspberry pi 4/5, but I'm sure they cannot handle even continuous gigabit data flow from Ethernet GMII interface. Just because much more powerful desktop PC cannot do it well without server ethernet adapter with hardware acceleration.

[Nominal Animal]

No, Rockchip RK3588S is even more powerful than RK3568.  These puppies have absolutely no problem throwing a gigabyte per second around, continously.

I didn't tried that RK3588S, but a much powerful desktop i3 has performance issues even with 20 MS/s.

I think you have way too much faith in Intel.

if you're planning a real usage through GMII or USB interface, it will starts packet drops at much slower data flow.

Sigh.  The USB 3 numbers are from real benchmarks.

Yes, Raspberry Pi's USB implementation is utter shite, because it can drop packets silently.
Also, Windows is not useful for this kind of stuff; you don't run a GUI on a high-performance measurement instrument.  The amount of resources Windows needs is not related to how much resources the same task in e.g. Linux needs, really.

UDP/IP payload bandwidth at optimal conditions using gigabit is about 120 MBytes/second, and Odroids can sustain about 90% of that, or 110 MBytes/second, as one can easily benchmark (including using simple C programs).  With a direct link, increasing MTU will give slightly better performance.

Radxa Rock5B, for example, has a 2.5Gbit Ethernet (RTL8125B), with practical bandwidth around 2.35 Gbit/s according to iperf benchmarking, so using UDP/IP over a dedicated link (no collisions), you should be able to get 2 Gbit/s or 250 Mbytes/s transfer rate in practice – which is two thirds more than you actually need.

And note that there is required continuous transfer in realtime, not a signle burst transfer.

I know.  One of my hobbies is interfacing to various sensory things using Teensy 4, a $24 dev board for NXP i.MX RT1062.  Unlike many other microcontrollers, this one has native USB 2.0 high speed interface (two, actually), and even a trivial USB Serial example can reach 200+ Mbit/s (or 25 Mbytes/s) payload in one direction.  I test this with a Xorshift64 PRNG, generating the data (it runs at 600 MHz) on both the Teensy and my host computer, so I even verify the data is correct.  I measure the interval between received chunks (using wall clock, or POSIX CLOCK_REALTIME in Linux), and update min, max, avg, and stddev (median for limited runs).  Then, I run it for a few hours to verify this can be sustained.
To get closer to the theoretical maximum, I need to use bulk transfers, but as that rate is more than my sensor and small display things need, I haven't bothered even testing the maximum.

I've also done server-side development for years, so I know what bandwidth, PPS, latency, and other related terms actually mean.

What I don't know, is the electronics side.  I'm a hobbyist newbie when it comes to electronics, especially things like how to implement proper LVDS (or MIPI CSI) interfaces at 1.2 Gbit/s bandwidth.

You need large buffers, preferably mostly continuous ones, filled/drained via DMA or a dedicated CPU, to reach 2+ Gbits/s throughput.
The memory bandwidth is about 6 Gbytes/sec ("48 Gbit/s") on Odroid-M1 (report), more on Rock5B and Orange Pi 5.

Storage tends to be quite optimized in Linux, so no problem there.  The main problem is obtaining the ADC data.  You will not achieve anywhere near the necessary bandwidth using GPIO pins; you'll need a native interface with DMA and sufficient bandwidth to do so.

In the aforementioned SBCs –– unlike e.g. Raspberry Pi's –– the storage buses (SATA and PCIe) are separate from the USB 3.  They are peripherals integrated to the SoC, with DMA access to the SoC memory.  While I haven't verified it, USB3 should easily exceed the needed bandwidth, if you had an USB 3 ADC or DAQ.  Both RK3588 and RK3568 MIPI CSI can do 2.5 Gbits/sec per lane, and the boards expose either two or four lanes, so MIPI CSI bandwidth would not be an issue if you can just get the ADC to talk MIPI CSI (assuming a DIY frontend-ADC-FPGA-MIPI solution).

I wish I had an Odroid-M1, Rock5B, or Orange Pi 5, because then I could actually easily verify this, simply by connecting USB3 to a JMS567-SATA bridge and to a fast SATA drive, and check the sustained transfer rates to/from a Samsung PCIe M.2 (or other similar SSD with sufficiently high bandwidth on PCIe 3.0).
I'm very sure the result is well over 150 Mbytes/second, probably twice that (unless limited by the SATA drive), based on my experience with other SBCs.
It is very much equivalent to using USB 3 bulk transfers to an userspace process, and that process storing it to the PCIe M.2 (or vice versa).

I'm sure [odroid-m1] cannot handle even continuous gigabit data flow from Ethernet GMII interface. Just because much more powerful desktop PC cannot do it well without server ethernet adapter with hardware acceleration.

You're utterly, utterly wrong.  Desktop PCs, especially those running Windows, aren't a good benchmark.

Look at the Rock5B iperf benchmark here.  This is just a short run, over 27 seconds, with at least 280 megabytes transferred each second (except the initial second).  I prefer at least 15-20 minute runs, myself, but there is no reason to think this cannot be sustained.

Even old Odroid XU4 can sustain 225 Mbytes/sec over USB3 using JMS578 USB-to-SATA bridge and Samsung SSD 850 PRO 256GB.  That is 1.8 Gbits/s, well over the 1.5 Gbits/s you need.  (I have the Odroid HC1 model, with the JMS578 bridge on board; on most SATA SSDs I have, the SSD itself is the bottleneck.)

[larsdenmark]

I don't see a need for continuous transfer. The OP states that he want to store data that are collected for a minute. Which means you can simply store the data in RAM and decide where to move the data afterwards.

The Orange Pi 5 plus has 2.5 GB ethernet interfaces so you can transfer the data quickly to somewhere else. I haven't tested this, but I would assume that it comes with these interfaces because it is able to move data that fast.

[radiolistener]

UDP/IP payload bandwidth at optimal conditions using gigabit is about 120 MBytes/second, and Odroids can sustain about 90% of that, or 110 MBytes/second, as one can easily benchmark (including using simple C programs).  With a direct link, increasing MTU will give slightly better performance.

As I remember, my desktop PC reports about 900 Mbps transfer speed in benchmark. But I can't get stable continuous realtime transfer without packet drops even at 600 Mbps.

Another example is my gigabit router DIR-655, it has gigabit ports, and peak transfer speed really going to about 1 Gpbs. But when I push realtime continuous stream into it, it just starts to drop packets at about 200-300 Mbps, when I increase transfer speed up to 700-900 Mbps it just leads to denial of service. But it has real gigabit ports and can transfer at gigabit speed... but this is not the case when stream is continuous and don't allows delays and waits...

When I tried to push 700-900 Mbps continuous UDP stream directly into gigabit ethernet port of my desktop PC running on Win 7, and there is no running program and no listener on UDP port, it leads to about 30-40% CPU load for 2 core i5 CPU which running at 3200 MHz (overclocked). That 30-40% CPU load is just to receive packets with OS driver at fully loaded gigabit link. This is why I don't believe that such a toy singleboard PC can handle it with no problem 

[ve7xen]

Generally it is pps that matters more than bps for realtime throughput. If you are sending ~1500b payloads, then any non-broken implementation should easily support 900mbps without drops. If you are trying to do it with 64b (minimum) size packets, (24x the overhead), it becomes a lot more difficult as the ISRs must be serviced more frequently to avoid drops in most designs, and obviously it is a lot more parsing and so on overhead in the OS.

I could believe that a reasonably modern machine might struggle with 1Gbps @ 64b packets, but there is no way that any remotely modern machine, even a cheap SBC, should struggle with 1Gbps @ 1500b packets. Notwithstanding whatever the processing / storage of that data requires, of course.

Realtek is known for quite poor drivers as well as hardware that requires relatively a lot of CPU involvement. Good NICs are not expensive, though, and the MACs in embedded CPUs tend to be fine as well.

[radiolistener]

Just found screenshot with my tests at about 935 Mbps. The receiver program just receive UDP packets, and check it's first 4 bytes with packet id to detect packet loss and calculate transfer speed per second. There is no other tasks and no background processes.

As I remember I was used UDP packet size 2048 bytes.

As you can see so easy task requires at least 46% CPU load for i5 @ 3200 MHz.
I just wonder how you can get better result on a much slower toy-machine?

[Nominal Animal]

This is why I don't believe that such a toy singleboard PC can handle it with no problem 

You believe exactly what you want.  However, even the older Linux SBCs I have, can stuff a gigabit ethernet full of data with no problems.

I even included links to real-world benchmarks (using iperf and other tools) on said hardware, but apparently your belief in the superiority of Windows is unshakeable even when shown contrary real-world results.

There is a reason why Windows is only used on the desktop, and why none of the worlds top 500 high-performance computers run Windows (they all run Linux, and have for years now), and even Microsoft has switched to using Linux outside their desktop business.
To me, Windows is the desktop-only toy.

[radiolistener]

Realtek is known for quite poor drivers as well as hardware that requires relatively a lot of CPU involvement. Good NICs are not expensive, though, and the MACs in embedded CPUs tend to be fine as well.

well, may be this is because realtek MAC chip, but I never seen a project for raspbery pi level machine that allows to get realtime stream from high speed ADC at rate more than 1-2 MS/s. And this is not because this is no needed. That ability is very interesting.

And high speed SDR receivers with USB3 interface that allows to receive at 20 MS/s and more write notice that such a speed requires modern top level PC, at least i5 or i7. And you can read feedback from users that it has performance issues on a slow machines at rate 10 MS/s and above.

[Nominal Animal]

Realtek is known for quite poor drivers as well as hardware that requires relatively a lot of CPU involvement. Good NICs are not expensive, though, and the MACs in embedded CPUs tend to be fine as well.

well, may be this is because realtek MAC chip, but I never seen a project for raspbery pi level machine that allows to get realtime stream from high speed ADC at rate more than 1-2 MS/s. And this is not because this is no needed. That ability is very interesting.

And high speed SDR receivers with USB3 interface that allows to receive at 20 MS/s and more write notice that such a speed requires modern top level PC, at least i5 or i7. And you can read feedback from users that it has performance issues on a slow machines at rate 10 MS/s and above.

Raspberry Pis have unreliable-hardware USB implementation.  They are not a valid choice for something like this.

As to the rest, you should note that that is your unfounded belief that others who actually use these SBCs disagree with, never having tested one, instead of positing it as a fact you have verified yourself.  Windows machines do not compare to SBCs, it's that simple.

[radiolistener]

I even included links to real-world benchmarks (using iperf and other tools)

I tested iperf. The problem with iperf is that it doesn't use continuous realtime stream at constant hardware controlled speed for measurements. It has a lot of random delays in transfer. In such relaxed mode you can transfer data at pretty high rate, but with no guarantee for a constant transfer speed. The actual speed is varying from low to high and iperf just measure average speed.

This is why iperf can show you nice digits, but these digits don't show you a real ability to transfer realtime stream at constant high speed close to the maximum limit of the gigabit link.

When you're trying to transfer realtime continuous stream at constant speed which is needed to transfer data from high speed ADC, it leads to a problems with packet drops on PC side. This is because such a mode requires guaranteed OS response within pretty low time interval. And higher speed means that this interval should be lower.

If CPU doing some non critical stuff and didn't processed packet within hardly defined time interval, it just drops the packet and it leads to malfunction, because there is no way to re transmit the packet.

Can you please show me example of high speed ADC data transfer implementation for a raspberry pi at rate about 10-20 MS/s and above that? I never hear that... The only implementation that I know which allows 20-40 MS/s transfer through USB3 requires top level desktop machine with i5/i7 CPU...

It will be great to have something like raspberry pi which allows to handle so fast realtime streams, but unfortunately I don't know such solution. You're just use desktop PC to handle stream at gigabit speed, or use raspberry pi and accept it's limit for a speed transfer to about 1-2 MS/s or even smaller.

By the way, I tried linux (Arch+plasma) and windows (Win7) comparison for SDR processing with SDRAngel and SDR++, I can say that I was expected that linux will be better for that. But in reality I found that linux implementation is a little bit slower and buggy. In addition I found that windows implementation works more smooth and stable. That was a surprise for me. 

Of course it's doesn't means that windows IP stack is better, but if we take all things together (IP stack, OpenGL, task scheduler and other things) the windows OS still win in that fight. 

As to the rest, you should note that that is your unfounded belief that others who actually use these SBCs disagree with

My belief is based on the fact that there is no even a single implementation for SBC to get realtime high speed ADC stream into SBC and process it in realtime without packet drops. Desktop PC has such implementation up to 40 MS/s and more.

I tested some different machines with aim to try implement such transfer, but unfortunately slow machines (like SBC) have big performance limitations which prevents to use realtime streams more than 1-2 MS/s. Maybe modern SBC have better performance and can do more, but 14-bit 75 MS/s realtime stream is too much even for a modern top performance desktop PC.

As I understand, your belief that SBC can handle 14-bit 75 MS/s realtime stream is based on iperf results. Well, but why no one can do it if it's possible?

I'm really interesting, maybe I don't know something and it's really possible, then can you show me some example? For example maybe there is some project that receive realtime ADC data at 40-75 MS/s and can process it in realtime. At the moment, with results that I was seen on different machines, I just don't have idea on how it can be possible on so slow SBC.