Author Topic: Easiest way to collect some data @50MB/s and dump to computer  (Read 10280 times)

0 Members and 1 Guest are viewing this topic.

Offline c64Topic starter

  • Frequent Contributor
  • **
  • Posts: 311
  • Country: au
Everything below is just for testing a prototype, i.e. not needed in production. It means solution can be ugly.

I receive some data at ~ 50 MBytes / sec into my FPGA (Cyclone IV E). I want to dump it to the computer. Not necessary in real time. Acceptable solution would be to collect just ~1 sec of data and then send to computer at slower speed.

Can you recommend some very easy way to do it? I have no experience with USB or SDRAM. I have USB logic analyzer,  8-bit 24MHz.

So far the only idea I have is to dump everything to memory (but 50MB of non-SDRAM is quite expensive) , then transfer to computer at slower speed using:
1. Logic analyzer (if it has this feature "dump everything to file")
2. LPT port :)
3. Some USB-to-UART thing
4. Write to SDCARD - but need to move it between DUT and computer all the time

I guess the proper thing would be to use USB 2.0 which should be good enough for this speed but have no experience with USB
« Last Edit: June 12, 2020, 06:53:56 am by c64 »
 

Offline ataradov

  • Super Contributor
  • ***
  • Posts: 11906
  • Country: us
    • Personal site
Re: Easiest way to collect some data @50MB/s and dump to computer
« Reply #1 on: June 12, 2020, 07:04:08 am »
Cypress CY7C68013 is a high-speed USB to FIFO interface that can do close to 50 MB/s. In theory USB HS can go slightly higher, but in reality I could never get more than around 48-49 MB/s from USB HS. I'm pretty sure this is OS overhead. Windows is slightly worse than Linux in this respect.

That Cypress chip will require some programming of 8051 core, but it is pretty trivial, and the task is so common that you will probably be able to find the readily available firmware.

There is equivalent chip from FTDI, which may not require programming, but then you will have to deal with FTDI.
« Last Edit: June 12, 2020, 07:05:43 am by ataradov »
Alex
 

Offline palpurul

  • Regular Contributor
  • *
  • Posts: 174
  • Country: tr
  • Hey
Re: Easiest way to collect some data @50MB/s and dump to computer
« Reply #2 on: June 12, 2020, 07:24:17 am »
I believe FT601 can achieve 50MB/s
 

Offline ataradov

  • Super Contributor
  • ***
  • Posts: 11906
  • Country: us
    • Personal site
Re: Easiest way to collect some data @50MB/s and dump to computer
« Reply #3 on: June 12, 2020, 07:26:15 am »
It is not about what device can do. They all can in theory. OSes can't handle 50 MB/s. Or at least I could not make them do it with many different USB devices, so I'm pretty sure it is OS limitation.

I used libusb on all of them, so it may be libusb thing, but I don't think so.
Alex
 

Offline palpurul

  • Regular Contributor
  • *
  • Posts: 174
  • Country: tr
  • Hey
Re: Easiest way to collect some data @50MB/s and dump to computer
« Reply #4 on: June 12, 2020, 07:33:27 am »
50 MB/s is pretty decent speed for USB HS. That's about the limit of the interface.
 

Offline ataradov

  • Super Contributor
  • ***
  • Posts: 11906
  • Country: us
    • Personal site
Re: Easiest way to collect some data @50MB/s and dump to computer
« Reply #5 on: June 12, 2020, 07:40:20 am »
But you can't actually get exactly 50 MB/s. So if the stream is really a fixed 50 MB/s stream, then USB is out of the question.
Alex
 

Offline palpurul

  • Regular Contributor
  • *
  • Posts: 174
  • Country: tr
  • Hey
Re: Easiest way to collect some data @50MB/s and dump to computer
« Reply #6 on: June 12, 2020, 07:42:22 am »
But you can't actually get exactly 50 MB/s. So if the stream is really a fixed 50 MB/s stream, then USB is out of the question.

Do you mean you can't sustain 50MB/s speed for a long time?
 

Offline magic

  • Super Contributor
  • ***
  • Posts: 7454
  • Country: pl
Re: Easiest way to collect some data @50MB/s and dump to computer
« Reply #7 on: June 12, 2020, 08:18:01 am »
Not even for a short time.

In my experience it was even slower, at some 45MB/s. You could use a pair of FX2 to double throughput. This will certainly work, as long as you use USB ports driven by different host controllers.

Otherwise you are looking at USB3 (no idea if anything cheap exists) or buffering some fraction of the data on your board. If FX2 can do 45MB/s, you only need to buffer the remaining 5MB of each burst.

You don't need to program the 8051 core on the FX2, there are ready firmwares just for that written by others. Your logic analyzer might contain an FX2.

LPT or UART - forget it. The fastest USB UART receivers might go to 12Mb/s. You would need to parallel quiet a few.
As for other ideas, I wonder how fast are GPIOs on those single board computers like the Pi and others.
« Last Edit: June 12, 2020, 08:22:22 am by magic »
 

Offline palpurul

  • Regular Contributor
  • *
  • Posts: 174
  • Country: tr
  • Hey
Re: Easiest way to collect some data @50MB/s and dump to computer
« Reply #8 on: June 12, 2020, 08:33:20 am »
You are correct about the UART. UART's going to be painfully slow. You gotta go with the parallel interface.

FT601 is about 9 dollars, but building a 4-layer PCB + impedance control is going to dominate the cost. Cypress has USB 3 micro's and they start from about 25 $ and they come with BGA packages you also need to build a 4-layer PCB, pay for impedance control, pay for assembly (suppose you can't hand solder bgas?  :-//) or you can buy their dev board which is not going to be cheap and can be useless depending on the application.
« Last Edit: June 12, 2020, 08:34:58 am by palpurul »
 

Offline ataradov

  • Super Contributor
  • ***
  • Posts: 11906
  • Country: us
    • Personal site
Re: Easiest way to collect some data @50MB/s and dump to computer
« Reply #9 on: June 12, 2020, 04:23:42 pm »
Do you mean you can't sustain 50MB/s speed for a long time?
I meant that USB HS device can't even peak at 50 MB/s.

But if FT601 is USB 3.0, then this is not a problem. But you would need to deal with USB 3.0, so your selection of ports and hardware becomes more limited.
Alex
 

Online SiliconWizard

  • Super Contributor
  • ***
  • Posts: 15802
  • Country: fr
Re: Easiest way to collect some data @50MB/s and dump to computer
« Reply #10 on: June 12, 2020, 04:46:10 pm »
Of course an USB-UART solution would likely be the easiest, but don't expect very high speeds.  Max speed with FTDI devices (and I don't think you'll get higher with competitors) is something like 3Mbits/s, which will yield something like 300KB/s tops.

If you want better throughput, IME a FT232H or FT2232H in parallel mode would be the fastest way to get there.
Real-world data: in asynchronous mode ("FT245" mode), you're going to get something like 6-8MB/s max. Simple to interface, may be enough for your needs. In synchronous mode, you're going to realistically get something like 30MB/s max, which is not too shabby. Interfacing in synchronous mode is a bit more involved though, and will usually mean having to deal with an additional clock domain in your FPGA.

For even faster speed, USB SS (for instance with FTDI's FT6xx offering) will get you much higher throughput, but if you're looking for an easy solution, the former IMO will be much easier and cheaper. You can find modules/breakout boards for FT232H or FT2232H for pretty cheap, and easy to use. I couldn't find such boards at the moment for FT60x chips, and the official dev boards are more expensive (~$100), and have connectors that may not be easy to interface with depending on your own board (such as FMC.)

Of course as someone suggested, you can make your own boards for this, but that's going to be some additional dev time and is going to definitely cost you more than buying dev boards or modules.

Yet another option would be to embed some soft ethernet controller in your FPGA and add an external ethernet PHY+connector (you can find ready-made boards with that.) I believe some people here have experience doing this, but it's going to be more involved than the above IMO.
 

Offline ledtester

  • Super Contributor
  • ***
  • Posts: 3284
  • Country: us
Re: Easiest way to collect some data @50MB/s and dump to computer
« Reply #11 on: June 12, 2020, 04:54:11 pm »
See if your logic analyzer is compatible with sigrok:

https://sigrok.org/

sigrok has command line tools which you can use to capture data directly to a file.
 
The following users thanked this post: iwasz

Offline Bassman59

  • Super Contributor
  • ***
  • Posts: 2501
  • Country: us
  • Yes, I do this for a living
Re: Easiest way to collect some data @50MB/s and dump to computer
« Reply #12 on: June 12, 2020, 08:41:44 pm »
There is equivalent chip from FTDI, which may not require programming, but then you will have to deal with FTDI.

The FTDI parts don't allow you to customize all of the device descriptor, so it can't enumerate as a standard device class such as Audio. That greatly limits its applicability.
 

Offline c64Topic starter

  • Frequent Contributor
  • **
  • Posts: 311
  • Country: au
Re: Easiest way to collect some data @50MB/s and dump to computer
« Reply #13 on: June 13, 2020, 03:41:21 am »

Thanks guys, I think I will abandon my ugly idea from the first post of having a memory "buffer".

So it will be all in real time like this: streem of data -> fpga -> some USB 2.0 device -> computer

Now just need to choose "some USB device".

I did some caclulations and data rate will be not more than 45MB/s. With some simple compression inside FPGA I can make it ~42MB/s if needed.

FT232H / FT2232H look very easy to use, but they say maximum speed is 40MB/s, so I think I will go with Cypress FX2 (CY7C68013)

Found quite cheap development board https://banggood.com/EZ-USB-FX2LP-CY7C68013A-USB-Core-Board-Development-Board-Logic-Analyzer-p-1183409.html so no need to make pcb
 

Offline ataradov

  • Super Contributor
  • ***
  • Posts: 11906
  • Country: us
    • Personal site
Re: Easiest way to collect some data @50MB/s and dump to computer
« Reply #14 on: June 13, 2020, 03:50:42 am »
Yep, I have a couple of those boards and they are fine.  And getting 45 MB/s should not be a problem at all.

Again, the only mildly annoying part about it is the need to program the MCU part. And of course you will have to make some PC software. So it is quite a bit of development effort.
Alex
 

Offline c64Topic starter

  • Frequent Contributor
  • **
  • Posts: 311
  • Country: au
Re: Easiest way to collect some data @50MB/s and dump to computer
« Reply #15 on: June 13, 2020, 04:13:16 am »
And of course you will have to make some PC software.
Yep. As I understand it's as simple as opening a stream and just reading data from it?
 

Offline ataradov

  • Super Contributor
  • ***
  • Posts: 11906
  • Country: us
    • Personal site
Re: Easiest way to collect some data @50MB/s and dump to computer
« Reply #16 on: June 13, 2020, 04:17:56 am »
Yep. As I understand it's as simple as opening a stream and just reading data from it?
It is quite easy with libusb, but not entirely trivial. There are enough examples though, and I can provide some help too.

On windows you would also need to make a fake driver, since windows is stupid. I usually use this tool https://zadig.akeo.ie/ It creates a nice complete driver package.
Alex
 

Offline c64Topic starter

  • Frequent Contributor
  • **
  • Posts: 311
  • Country: au
Re: Easiest way to collect some data @50MB/s and dump to computer
« Reply #17 on: June 13, 2020, 04:34:19 am »
Darn really. Even need to make drivers. FTDI is so much easier
 

Offline ataradov

  • Super Contributor
  • ***
  • Posts: 11906
  • Country: us
    • Personal site
Re: Easiest way to collect some data @50MB/s and dump to computer
« Reply #18 on: June 13, 2020, 04:37:15 am »
It is not really "making". You just press a couple buttons in that tool and it makes them for you. It also changes descriptor strings and all that stuff, so your device is recognized by the system and looks all professional.

From the actual programming point of view, I don't know what is easier, I have not programmed anything FTDI, but from what I've seen, I prefer libusb.
Alex
 

Offline magic

  • Super Contributor
  • ***
  • Posts: 7454
  • Country: pl
Re: Easiest way to collect some data @50MB/s and dump to computer
« Reply #19 on: June 13, 2020, 04:59:41 am »
AFAIK for FTDI you can just use a similar libftdi where you open_device(), configure_device() and read_device(), roughly speaking.

I wonder if the dual FT2232H would be capable of simultaneously running two streams with its two engines and achieving your throughput target in aggregate.
 

Offline ataradov

  • Super Contributor
  • ***
  • Posts: 11906
  • Country: us
    • Personal site
Re: Easiest way to collect some data @50MB/s and dump to computer
« Reply #20 on: June 13, 2020, 05:07:32 am »
Well, after all the enumeration and device search libusb is the same. Usually for higher performance you submit multiple transfer requests and the library calls a callback function when the request is fulfilled. In the callback after processing the data you can re-submit the same request. This way you always have some pending requests, so no wasted bandwidth.
Alex
 

Offline c64Topic starter

  • Frequent Contributor
  • **
  • Posts: 311
  • Country: au
Re: Easiest way to collect some data @50MB/s and dump to computer
« Reply #21 on: June 13, 2020, 05:08:44 am »
AFAIK for FTDI you can just use a similar libftdi where you open_device(), configure_device() and read_device(), roughly speaking.

This is how understand as well from this document
https://www.ftdichip.com/Support/Documents/AppNotes/AN_130_FT2232H_Used_In_FT245%20Synchronous%20FIFO%20Mode.pdf

No need to write firmware or generate drivers with some third party tool

I wonder if the dual FT2232H would be capable of simultaneously running two streams with its two engines and achieving your throughput target in aggregate.
I think dual in it means "dual UART"
 

Offline ataradov

  • Super Contributor
  • ***
  • Posts: 11906
  • Country: us
    • Personal site
Re: Easiest way to collect some data @50MB/s and dump to computer
« Reply #22 on: June 13, 2020, 05:11:52 am »
You don't have to use that tool. You can write INI file yourself if you want to. That's just how windows is, you need a "driver", even if it is just a dummy.

FTDI obviously provides their own driver, but you still have to install that. There is not a whole lot of difference.
Alex
 

Offline c64Topic starter

  • Frequent Contributor
  • **
  • Posts: 311
  • Country: au
Re: Easiest way to collect some data @50MB/s and dump to computer
« Reply #23 on: June 13, 2020, 05:39:29 am »
You don't have to use that tool. You can write INI file yourself if you want to. That's just how windows is, you need a "driver", even if it is just a dummy.

FTDI obviously provides their own driver, but you still have to install that. There is not a whole lot of difference.
Ah, so driver itself is already written by Sypress, you only write (or generate) inf file?
 

Offline ataradov

  • Super Contributor
  • ***
  • Posts: 11906
  • Country: us
    • Personal site
Re: Easiest way to collect some data @50MB/s and dump to computer
« Reply #24 on: June 13, 2020, 05:42:52 am »
Ah, so driver itself is already written by Sypress, you only write (or generate) inf file?
Not exactly. Cypress chip is just a bare chip, it does nothing without firmware, so Cypress can't make that driver.

The actual driver is written by libusb team, or by Microsoft (if you use WinUSB driver). And all that tool does is generate INF file and put all the relevant driver files into a single folder. You mostly don't need it for one-off project. But if you were to give that device to someone else, this folder would be "the drivers" for your device.

And yes, you can write your own INF file if you want. But the tool is nice and easy to use. It will automatically pick up your device VID/PID and all that stuff. It saves a lot of time trying to install and uninstall half baked hand written INF files.

Also starting from Win10 you can make stuff work with WinUSB driver without manual installation. I don't remember the details, but I think windows automatically installs WinUSB driver if your device responds to a specific string descriptor with some predefined GUID-like string.
« Last Edit: June 13, 2020, 05:46:42 am by ataradov »
Alex
 
The following users thanked this post: c64


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf