Streaming data FPGA interface -> off the shelf?

[drwho9437]

I am looking for good solutions to provide steaming data back to a computer from a digital system.

I am aware of Opal Kelly's offerings, but feel the value proposition for me isn't that great. I don't need a big FPGA or lots of memory really and of course they realize the value of their software and charge a reasonable industrial rate for their stuff. At work that is fine but not for home stuff.

I have thought about using the FX-3 USB 3 chip myself and making my own but of course the development time would be high. It is also only BGA. FTDI has some USB 3 FIFO chips that look possible too but again lots of driver work and more to get it going.

So bottom line: what are the options out there? Seems like this should be a solved problem with a low margin off the shelf option -> a fat low latency pipe from a computer to an FPGA.

What do you use?

[Foxxz]

I was going to suggest the FX3 until you ruled it out. Not sure how much driver work needs to be done as I've seen these paired with plenty of FPGAs.

I might suggest an FTDI FT2232H which is USB2 but the driver work would be little to nothing.

There are FPGA PCIE bridges out there for which the driver is free but you license the IP to use in your design. If you need it to be detachable I have gotten it to work over lightning ports.

[SiliconWizard]

I might suggest an FTDI FT2232H which is USB2 but the driver work would be little to nothing.

Yes, drivers are available for all major platforms, plus a decent API.
They absolutely have USB 3 SS chips these days. FT6xx series. Not only are they no work for the drivers, but they are also significantly cheaper than the FX3.

[drwho9437]

I was going to suggest the FX3 until you ruled it out. Not sure how much driver work needs to be done as I've seen these paired with plenty of FPGAs.

I might suggest an FTDI FT2232H which is USB2 but the driver work would be little to nothing.

There are FPGA PCIE bridges out there for which the driver is free but you license the IP to use in your design. If you need it to be detachable I have gotten it to work over lightning ports.

I'm not ruling out the FX3. It seems like a lot of people have selected it so it must have something going for it but I'm not that strong on USB driver classes. FTDI has a USB 3 chip that is a hard wired FIFO of some kind in a somewhat easier to solder QFN than the FX3 BGA. Not sure if there is a serious drawback to the FTDI solution.

[drwho9437]

I might suggest an FTDI FT2232H which is USB2 but the driver work would be little to nothing.

Yes, drivers are available for all major platforms, plus a decent API.
They absolutely have USB 3 SS chips these days. FT6xx series. Not only are they no work for the drivers, but they are also significantly cheaper than the FX3.

Have you use the FT6xx series? I looked at the docs but it wasn't clear exactly what the PC interface would look like via the standard driver. Development board is about 100 dollars and I considered getting it and the cyclone dev board it mates to just to figure out how to use everything that way. Probably sell the cyclone board after I understand everything was my first line of thinking.

Wonder if there are other options with higher levels of integration though like the Opel Kelly stuff but less expensive.

[SiliconWizard]

I might suggest an FTDI FT2232H which is USB2 but the driver work would be little to nothing.

Yes, drivers are available for all major platforms, plus a decent API.
They absolutely have USB 3 SS chips these days. FT6xx series. Not only are they no work for the drivers, but they are also significantly cheaper than the FX3.

Have you use the FT6xx series? I looked at the docs but it wasn't clear exactly what the PC interface would look like via the standard driver.

Not yet, but you can read the doc here: https://ftdichip.com/wp-content/uploads/2020/08/AN_379-D3xx-Programmers-Guide-1.pdf

I have used the USB 2 HS parts a lot though, with their D2XX drivers, and the API looks absolutely similar.
Pretty easy to use.

[asmi]

Have you use the FT6xx series? I looked at the docs but it wasn't clear exactly what the PC interface would look like via the standard driver. Development board is about 100 dollars and I considered getting it and the cyclone dev board it mates to just to figure out how to use everything that way. Probably sell the cyclone board after I understand everything was my first line of thinking.

I have used FT601 with Artix-7 FPGA, and was very happy with the results. They provide a library for PC side (along with drivers of course) which you can use in C/C++ or .NET, I used the latter and was able to reach about 325 MBytes/s of sustained data bandwidth (that's the real speed your application is receiving data at after all overheards are accounted for). On FPGA side it's just a simple FIFO-style interface. I used it on my own custom board, so not sure how it works on their devboards. The only real challenge is figuring out a way to route so many parallel traces to FPGA without getting crosstalk issues. If you don't need that much of a bandwidth, you might want to use FT600 instead as it uses only 16bit wide data bus (as opposed to 32bit for FT601).

This is as "off the shelf" as it goes in this business, FPGA side interface is not much more complex than interfacing with internal FIFO, you can choose the frequency it runs at between 66 and 100 MHz (I used the latter, the former will reduce available bandwidth), and it's just a few lines of code on the PC side (at least using their .NET library).

Ethernet is much more complicated solution than FT60x, not to mention if you do need something close to 2.5 Gbps which they provide. PCI Express is even worse as it requires using dedicated multi-gigabit transceivers, as well as writing a driver on the PC side. Xilinx provides a free IP and a driver, but it's only for Linux. other vendors charge big $$$ even for IP.

[SiliconWizard]

For the FT232H/2232H, the synchronous FIFO mode is a "master" FIFO - the FT part provides the clock. So either you can use the same clock throughout your design (but beware this clock will stop when the communication is closed), or you need to synchronize between clock domains. Not a big deal, just something to keep in mind.

I'm not sure about the FT6xx - I think they act as FIFO slave, thus you provide the clock, which makes things a little easier - but asmi can confirm.

[langwadt]

I might suggest an FTDI FT2232H which is USB2 but the driver work would be little to nothing.

Yes, drivers are available for all major platforms, plus a decent API.

yes, good for about 40MB/s in fifo mode and you can use the second interface to configure the fpga

[asmi]

I'm not sure about the FT6xx - I think they act as FIFO slave, thus you provide the clock, which makes things a little easier - but asmi can confirm.

No, it's a master too (meaning the chip provides the clock), but you are likely want to have FIFOs on FPGA side as well to cross into whatever clock domain your design is running at, so it's not a big deal.

[voltsandjolts]

cross-ref to another thread and an interesting post about fx2pipe for cypress fx2 (never tried it myself).

[NorthGuy]

It is huge difference in speed between USB 2 and USB 3. The USB 2 solutions are abundant. USB 3 options are very scarce. All I could find is Cypress FX3 and FT601. FX3 is expensive, complex, and poorly documented. So, I made a prototype board with FT601. It worked well. It took me less than a day to begin communicating with FPGA through their library.

The FT601 FPGA FIFO interface is rather straightforward. The timing requirements in their documentation are very strange. They require huge hold time (4.8 ns) which is absolutely impossible to meet. They have an XDL example where they "forgot" to put minus - they have "set_output_delay -min 4.8" instead of "set_output_delay -min -4.8". This certainly passes timing analysis every time. But if you put the "minus" in, it's utterly impossible. Even though I had some help from the trace delay (400 ps), I could only meet 2.3 ns hold. This worked well at 100 MHz, buf failed at 66 MHz. From which I concluded that 4.8 ns is either bogus, or reflects only the worst case (66 MHz at 1.8V).

The DLL works, but is cumbersome, loads slow (300 ms), and is somewhat buggy. FT601 doesn't really do FIFO, rather block transfers with FIFO interface to FPGA. If you do simple block transfers with blocks of the fixed size everything works. With variable block size it becomes less reliable. Sometimes the read function returns with data size less than requested (contrary to the docs). If application quits without reading all the requested data, next time the DLL loads, it can read leftover data from the previous start. Sometimes it simply hangs and starts returning errors on all reads.

I couldn't get full speed from the DLL (381 MB/s), even when reading one long block. No matter what you do there are delays between blocks, so you realistically get 320-340 MB/s. While analyzing delays on the FPGA side I noticed that gaps between blocks are typically small, but sometimes there are longer delays as well. The nature of these delays is unknown. May be FT601 limitations. May be DLL problems.

It should be possible to figure out the USB protocol directly and get rid of the DLL, fixing the bugs, and possibly improving performance. I was going to explore this opportunity, but by that time the cheap Xilinx FPGAs turned into unobtainium and the project was scrapped.

[drwho9437]

You mentioned "home stuff" prices.

So if you're not looking for something industrial and embeddable in production why not consider the aforementioned (Cypress, FTDI) USB3 bridges *developer kits*?  They're "ready to do" except for wiring interface to your target system (which itself is non-trivial for high speed signal integrity if you have NN high speed digital signals to attach, so making an interface PCB + cable assembly is a good idea).

Also why not just any common FPGA board which includes a gigabit ethernet interface?  There are several out there for moderate prices and then you have almost the simplest possible streaming connection to a PC if you send UDP or such like packets.  And there are free IP cores / codes etc. to do it.  If you spend more you can get ones with multiple 1Gb Ethernet interfaces or something like 10Gb/s ethernet interfaces copper or with a SFP cage for your transceiver of choice.

If you look at sigrok.org and "supported hardware" they have a list of "logic analyzer" type digital signal acquisition boards that are supported by the sigrok open source software for digital state capture at the specified rates and port counts.  Among those are the typical sort of inexpensive "hobbyist" boards as well as some mid-range and more commercial / industrial oriented stuff.

The nice thing about ethernet though is that you can have a choice of either using several gigabit ethernet links coming from one board (if needed) or using several different interface boards each with one 1Gb/s interface and then a choice of the switch / NIC on the PC side that can be 10Gb, 5Gb, 2.5Gb, 1Gb per port with N ports so the scalability of the solution in terms of multiplicity of targets as well as multiplicity of bandwidth is good.  And you can get cheap PC hardware and switches and even cheap-ish devkits with ethernet & FPGAs.

Then there's PCI where there are again some relatively low cost available PCI/PCIE devkits which can be used and there are IP cores / sample designs out there that make it easier to use.

I have certainly thought about going the Ethernet UDP route for a long time but I have dithered on it. PCIE is a no, I want the box on a cable. I don't think the dev kits are a problem I just want to know the market before I invest the time. Still not sure if there are any higher level of integration other than Opal Kelly. I agree though GigE would be lower cost and in some ways simpler.

[drwho9437]

Have you use the FT6xx series? I looked at the docs but it wasn't clear exactly what the PC interface would look like via the standard driver. Development board is about 100 dollars and I considered getting it and the cyclone dev board it mates to just to figure out how to use everything that way. Probably sell the cyclone board after I understand everything was my first line of thinking.

I have used FT601 with Artix-7 FPGA, and was very happy with the results. They provide a library for PC side (along with drivers of course) which you can use in C/C++ or .NET, I used the latter and was able to reach about 325 MBytes/s of sustained data bandwidth (that's the real speed your application is receiving data at after all overheards are accounted for). On FPGA side it's just a simple FIFO-style interface. I used it on my own custom board, so not sure how it works on their devboards. The only real challenge is figuring out a way to route so many parallel traces to FPGA without getting crosstalk issues. If you don't need that much of a bandwidth, you might want to use FT600 instead as it uses only 16bit wide data bus (as opposed to 32bit for FT601).

This is as "off the shelf" as it goes in this business, FPGA side interface is not much more complex than interfacing with internal FIFO, you can choose the frequency it runs at between 66 and 100 MHz (I used the latter, the former will reduce available bandwidth), and it's just a few lines of code on the PC side (at least using their .NET library).

Ethernet is much more complicated solution than FT60x, not to mention if you do need something close to 2.5 Gbps which they provide. PCI Express is even worse as it requires using dedicated multi-gigabit transceivers, as well as writing a driver on the PC side. Xilinx provides a free IP and a driver, but it's only for Linux. other vendors charge big $$$ even for IP.

Very helpful advice. I might try to build a custom Max10 + FT600 board and see how far I get.

[drwho9437]

It is huge difference in speed between USB 2 and USB 3. The USB 2 solutions are abundant. USB 3 options are very scarce. All I could find is Cypress FX3 and FT601. FX3 is expensive, complex, and poorly documented. So, I made a prototype board with FT601. It worked well. It took me less than a day to begin communicating with FPGA through their library.

The FT601 FPGA FIFO interface is rather straightforward. The timing requirements in their documentation are very strange. They require huge hold time (4.8 ns) which is absolutely impossible to meet. They have an XDL example where they "forgot" to put minus - they have "set_output_delay -min 4.8" instead of "set_output_delay -min -4.8". This certainly passes timing analysis every time. But if you put the "minus" in, it's utterly impossible. Even though I had some help from the trace delay (400 ps), I could only meet 2.3 ns hold. This worked well at 100 MHz, buf failed at 66 MHz. From which I concluded that 4.8 ns is either bogus, or reflects only the worst case (66 MHz at 1.8V).

The DLL works, but is cumbersome, loads slow (300 ms), and is somewhat buggy. FT601 doesn't really do FIFO, rather block transfers with FIFO interface to FPGA. If you do simple block transfers with blocks of the fixed size everything works. With variable block size it becomes less reliable. Sometimes the read function returns with data size less than requested (contrary to the docs). If application quits without reading all the requested data, next time the DLL loads, it can read leftover data from the previous start. Sometimes it simply hangs and starts returning errors on all reads.

I couldn't get full speed from the DLL (381 MB/s), even when reading one long block. No matter what you do there are delays between blocks, so you realistically get 320-340 MB/s. While analyzing delays on the FPGA side I noticed that gaps between blocks are typically small, but sometimes there are longer delays as well. The nature of these delays is unknown. May be FT601 limitations. May be DLL problems.

It should be possible to figure out the USB protocol directly and get rid of the DLL, fixing the bugs, and possibly improving performance. I was going to explore this opportunity, but by that time the cheap Xilinx FPGAs turned into unobtainium and the project was scrapped.

Very interesting hopefully you will unscrap it if the chip shortage eases. I think a well documented reference big pipe OSH design could benefit people a lot. In my mind something like a Max10 base able to pull some LVDS lines to some DAC/ADC would be a thing that would be pretty enabling.

I didn't completely follow your timing issue but I guess I would figure it out when I get there. I haven't made a comparison between the FT6xx and the older FT FIFO stuff. Other than the PHY is there much difference in using them? I'd rather standardize my design on something future looking even if the older USB 2 FIFO is a tiny bit easier. If it is a lot easier perhaps I should rethink.

[NorthGuy]

I didn't completely follow your timing issue but I guess I would figure it out when I get there. I haven't made a comparison between the FT6xx and the older FT FIFO stuff. Other than the PHY is there much difference in using them? I'd rather standardize my design on something future looking even if the older USB 2 FIFO is a tiny bit easier. If it is a lot easier perhaps I should rethink.

USB 2.0 full speed uses 480 MHz signaling. This will give you 50 MB/s at the very best. If this is sufficient for you, I would definitely go with USB 2. I don't know what you mean by "easier", but it's certainly much cheaper, uses slower signals, saves space on PCB, and gives you more options. If you need 50 MB/s or more, then USB 2 is not fast enough for you.

[Someone]

I think a well documented reference big pipe OSH design could benefit people a lot.

Pick your favorite open source SDR.

spoiler... they mostly use FX2/FX3 chips!

[laugensalm]

One more thumb up for all the GNU radio derivatives, they offer the best FX2 code I've seen so far. Similar usable reference designs are found for FX3 *without* the flaky ThreadX framework (flaky as in introducing spurious DMA timeouts).

However, I'd definitely go with UDP over GigE these days, the software overhead for USB host-side (drivers, VID/PID handling) plus embedded firmware is way over the little more expense for the ethernet streaming logic. Especially if out of band messages are to be transmitted, Ethernet does the better job. Depending on what you're streaming, you might also rather try compressing it (DPCM, context based prediction, etc.) instead of driving your interface near the maximum possible data rate.

[voltsandjolts]

they offer the best FX2 code I've seen so far

Can you provide a link to your recommended fx2 code please?

[laugensalm]

This is the most recent I've found (my repo clones are kinda outdated).

https://github.com/EttusResearch/uhd/tree/master/firmware/fx2

Also, you might find derived code around all 'usbjtag' related source, at least for bulk transfers. All this code should compile with the license-free SDCC, I just remember some hickups with the migration to SDCC v3.

[Morgan127]

I can recommend you to have a look at the Ethernet cores from FPGA-Cores.com.
These are very easy to use and you get UDP / TCP and other stuff like remote programming, logic analyzer etc.
No charge for private projects. Free to download. I use them in almost all my projects.

[NorthGuy]

I can recommend you to have a look at the Ethernet cores from FPGA-Cores.com.
These are very easy to use and you get UDP / TCP and other stuff like remote programming, logic analyzer etc.
No charge for private projects. Free to download. I use them in almost all my projects.

Looks like these are RMII (10/100Mbit Ethernet), which is even slower than USB 2. Various methods can be ranked by their speed as follows:

USB 3 > Gigabit Ethernet > USB 2 > 100Mbit Ethernet

[Morgan127]

I can recommend you to have a look at the Ethernet cores from FPGA-Cores.com.
These are very easy to use and you get UDP / TCP and other stuff like remote programming, logic analyzer etc.
No charge for private projects. Free to download. I use them in almost all my projects.

Looks like these are RMII (10/100Mbit Ethernet), which is even slower than USB 2. Various methods can be ranked by their speed as follows:

USB 3 > Gigabit Ethernet > USB 2 > 100Mbit Ethernet

There are MII, RMII, GMII and RGMII so also Gigabit Ethernet.

[drwho9437]

For industrial application GigE or other Ethernet is good but the switch is some overhead and power over Ethernet is a thing but not as simple. I am going to take a look at the FTDI 16-bit FIFO chips in a bit more detail and see how far I get I think. My preliminary pick is 10M02SCM153 + FT600 -> connector. Sometimes I don't have time to really get into this stuff so we will see how far I get. This work isn't my highest priority but its somewhere on the list.

[jevinskie]

You might like this Atrix 7 35k paired with a FX3: https://m.aliexpress.com/item/1000006618157.html