Capturing 64Bit Bus

[luky315]

I have a 64Bit Bus with 16MHz clock rate (the DataOut lines of 64 1MSPS 12Bit SPI ADCs plus the Chip Select / Conversion Start Signal and the common 16MHz clock signal)
Now I need to capture the data and send it to a PC for analysis. The Cypress FX3 plus some "glue logic" looks like a good starting point.
Or is there a better way? I think that the solution would looks a lot like a 64Bit logic analyzer with 16MS/s Sampling rate.

[tszaboo]

That is a lot of data. More than a gigabit/sec. Sending that much data into a PC, isochronous is very very demanding. In fact, see Cypress' report on the topic:
http://www.cypress.com/file/125281/download
They basically say, it is not possible.
So I guess all remains is to see, how can you buffer it. Or how to review the test setup.
DOH! FX3 does 2800 megabit/sec!

[bobaruni]

Why would you need to send the clock via USB as well?
If you are only interested in the data then you have 64 x 1MSPS x 12 bits which brings it down to 768 Megabits....ouch still a lot.
You could possibly do some minor lossless compression 1.8:1 or If the exact ADC sample value is not essential then you can achieve 4:1 compression using ADPCM but at these speeds, FPGA or a very very fast MCU or SoC will be required.
Yet another option if the signals aren't continues is to buffer them in RAM then slowly retrieve them via USB.

[luky315]

The "Conversion Start" signal is only used to synchronize the incomming signals and the clock would be used as a sample signal (falling edge)
The problem is that if I use some sort of compression or digital zoom (8 bit out of 12) I need an FPGA and I have not a lot of experience with them :-(

[bobaruni]

Yep, FPGA can be difficult, a fast MCU like STM32H7 @400 MHZ might be able to do compression if optimized very well or maybe XMOS xCore would be more suited.
What is the nature of the signal being sampled? Continuous/Periodic, Bandwidth, Max freq, precision requirement etc?

[james_s]

How much data do you need to capture? This sounds like a job for a logic analyzer, unless you need to capture a large amount of data in which case an FPGA is a good candidate. I like working with those because they're extremely flexible, you can change the hardware just by modifying some code. Even a cheap low end FPGA will have no trouble capturing at 16MHz and most will have more than enough IO to read a full 64 bit bus. You may need to allow more IO to interface with external RAM depending on how much data is coming out and where you're sending it.

[nctnico]

Without knowing how much data needs to be sampled in total it is impossible to propose a good solution.

[rstofer]

Without knowing how much data needs to be sampled in total it is impossible to propose a good solution.

Exactly!  An FPGA can easily keep up with the data rate but memory resources (BlockRAM) might be limited.  I don't know if the DDR3 core for an FPGA will keep up over the long haul.

But at some point, there needs to be an end to the traffic because it won't be long before a 1TB hard drive won't hold it.

I'm not sure a PC can keep up because, if USB won't work, what else is there other than building a custom PCI board?

How long is the capture?  Width isn't a problem and neither is rate but length is where things turn ugly.

[Rasz]

cmon guys, its only 130MB/s, number of IO is the real problem, you will need at least CPLD to grab all that data and pack it into neat 16/32bit bus for FX3
you could even push it over 4x FX2 if you have access to computer with 4 independent USB busses - modern desktop + 1-2 PCIE USB cards. laptops might be problematic.

[nctnico]

cmon guys, its only 130MB/s, number of IO is the real problem, you will need at least CPLD to grab all that data and pack it into neat 16/32bit bus for FX3
you could even push it over 4x FX2 if you have access to computer with 4 independent USB busses - modern desktop + 1-2 PCIE USB cards. laptops might be problematic.

If you only need to capture a short burst (less than several Mpts) then a logic analyser is infinitely easier because there is no hardware that needs to be developed.

[tszaboo]

I have no idea how I srewed up a simple calculation like that. FX3 does 360 MB/sec.

[Brutte]

I'd use 4x FT232H and 4 USB hosts. It is true USB Hi-Speed does 40MB/s but four USB hosts do much faster. You can buy FT232H for under $3. These support synchronous parallel bus. If you clock all of them with one clock, they won't miss a bit.

[CM800]

Why not into an FPGA then through to the computer VIA Ethernet UDP? you can get some cheap fpga dev kits that can do that.

[luky315]

Thanks for all the answers. But I think that I should clarify my requirements a bit:
The clock rate is 16 MHz and the Slave clocks out 14 Bit of data (2 Dummy bits and 12 Bit of "real" Data) to 64 parallel data lines after the #SS signal goes low (see attachment)
I don't have a problem using multiple devices as long as they are synchronizeable (only one clock and SS Signal)

[Rasz]

it it a continuous stream?
if packets whats the delay between?

[rstofer]

Thanks for all the answers. But I think that I should clarify my requirements a bit:
The clock rate is 16 MHz and the Slave clocks out 14 Bit of data (2 Dummy bits and 12 Bit of "real" Data) to 64 parallel data lines after the #SS signal goes low (see attachment)
I don't have a problem using multiple devices as long as they are synchronizeable (only one clock and SS Signal)

Your attachment describes a single frame.  How many frames per second?  How far apart are the frames?  Grabbing 64 streams, in parallel, is trivial with an FPGA (assuming sufficient pin count) but it's the rest of the data rate that matters.  As logic analyzers go, this clock rate is no big deal, the number of channels is no big deal but total record length might be an issue.

I would probably use either a CPLD or an FPGA to grab the serial streams and stuff the data into the FX3 over a parallel bus.  The FX3 has a DMA mechanism and a large amount of RAM that can act as a buffer.  The only challenge is whether the host PC can actually handle the incoming USB rate.

FPGA board with FX3S, SD card and DDR3 memory (pretty pricey)
https://www.ztex.de/usb-fpga-2/usb-fpga-2.14.e.html

Reference on FX3S including a book - I haven't studied the difference between the FX3 and the FX3S in terms of RAM and DMA
http://www.cypress.com/products/ez-usb-fx3s

[Rasz]

The only challenge is whether the host PC can actually handle the incoming USB rate.

no, nineties were ~20 years ago. Hundreds of MB/s are not a problem today, unless someone programs in python ;-). Modern computers can do things like sustained >500MB/s 2:1 compression using one core, and usually you have 4 cores to play with

btw "64 1MSPS 12Bit SPI ADCs" reminds me of something else - arrays of mems microphones for sound visualization, except those are continuous PDM stream.

[tszaboo]

The only challenge is whether the host PC can actually handle the incoming USB rate.

no, nineties were ~20 years ago. Hundreds of MB/s are not a problem today, unless someone programs in python ;-). Modern computers can do things like sustained >500MB/s 2:1 compression using one core, and usually you have 4 cores to play with

btw "64 1MSPS 12Bit SPI ADCs" reminds me of something else - arrays of mems microphones for sound visualization, except those are continuous PDM stream.

Then I encurage you to try a serial port at some 10 Mbit. Sure, with correct drivers it can handle 1Gbit on the ethernet and 1000MB/sec on the PCIexpress, but any kind of serial port will mess up your PC, using 50% of CPU, packet losses and so on. The magic is in the proper drivers, or the complete lack of it.

[Rasz]

thats why modern computers dont even have serial ports anymore There is no problem transferring 4GBit over USB 3.0. Sure, max USB 3.0 transfer it will eat whole cpu core to babysit, but you have 3-7 left. I think only with USB 3.1 controllers companies started to bother implementing DMA . ASM1142 can pump 10GBit at one core single digit utilization.

[james_s]

My 1.5 year old i7 desktop still has a serial port, and while I do still use it occasionally it wouldn't be my first choice for an application like this. Ethernet is probably the most flexible, it's a great way to move a very large amount of information, and sustained transfers are not a problem.

[tszaboo]

thats why modern computers dont even have serial ports anymore There is no problem transferring 4GBit over USB 3.0. Sure, max USB 3.0 transfer it will eat whole cpu core to babysit, but you have 3-7 left. I think only with USB 3.1 controllers companies started to bother implementing DMA . ASM1142 can pump 10GBit at one core single digit utilization.

Yes. Yet, you have devices , like the FT2232, which in theory can give you 12 Mbit UART or 40MByte FIFO interface. And with a small 80 MHz microcontroller you can totally kill a 3Ghz i7, flooding it with data, and driving that single core to 100% and getting data loss. Because drivers are written by the awesome programmers.
So any time, I hear sustained data through a not well implemented interface, red flags are raised in my mind.
Getting large transfer speeds is not the issue here. Getting it consistently is. That is why a USB display adapter is much more difficult and expensive than a USB SATA adapter.

[Rasz]

thats why modern computers dont even have serial ports anymore There is no problem transferring 4GBit over USB 3.0. Sure, max USB 3.0 transfer it will eat whole cpu core to babysit, but you have 3-7 left. I think only with USB 3.1 controllers companies started to bother implementing DMA . ASM1142 can pump 10GBit at one core single digit utilization.

Yes. Yet, you have devices , like the FT2232, which in theory can give you 12 Mbit UART or 40MByte FIFO interface. And with a small 80 MHz microcontroller you can totally kill a 3Ghz i7, flooding it with data, and driving that single core to 100% and getting data loss. Because drivers are written by the awesome programmers.
So any time, I hear sustained data through a not well implemented interface, red flags are raised in my mind.
Getting large transfer speeds is not the issue here. Getting it consistently is. That is why a USB display adapter is much more difficult and expensive than a USB SATA adapter.

this is neither uart nor USB limitation, you just chose shit usb to serial converter(or rather set it up in bad way) that doesnt buffer data and instead sends individual bytes(sure hope its not bits) in USB Isochronous mode as individual packets (one packet is like 3 transactions with handshake and everything) - that is insane.

[tszaboo]

this is neither uart nor USB limitation, you just chose shit usb to serial converter(or rather set it up in bad way) that doesnt buffer data and instead sends individual bytes(sure hope its not bits) in USB Isochronous mode as individual packets (one packet is like 3 transactions with handshake and everything) - that is insane.

Sure.

Dual Port TX Buffers (128 bytes): Data from the USB data out endpoint is stored in the Dual Port TX
buffer and removed from the buffer to the transmit register under control of the UART FIFO controller.
Dual Port RX Buffers (384 bytes): Data from the UART / FIFO controller receive register is stored in the
Dual Port RX buffer prior to being removed by the SIE on a USB request for data from the device data in
endpoint.

Since you obviously have no idea what you are talking about, I consider this discussion closed.

[Rasz]

this is neither uart nor USB limitation, you just chose shit usb to serial converter(or rather set it up in bad way) that doesnt buffer data and instead sends individual bytes(sure hope its not bits) in USB Isochronous mode as individual packets (one packet is like 3 transactions with handshake and everything) - that is insane.

Sure.

Dual Port TX Buffers (128 bytes): Data from the USB data out endpoint is stored in the Dual Port TX
buffer and removed from the buffer to the transmit register under control of the UART FIFO controller.
Dual Port RX Buffers (384 bytes): Data from the UART / FIFO controller receive register is stored in the
Dual Port RX buffer prior to being removed by the SIE on a USB request for data from the device data in
endpoint.

Since you obviously have no idea what you are talking about, I consider this discussion closed.

I will happily admit I have no clue about FTDI, but here I am transferring 40MB/s using Cypress FX2 with 5% cpu load = your problem is caused by you yourself, and has nothing to do with USB.

[james_s]

Or caused by the driver developers, most of us do not develop our own drivers, we are at the mercy of hardware manufactures and the drivers they supply. My own observation is that it is highly variable, some stuff works great, other stuff doesn't. I have never been able to get my Bitscope, MiniLA or Open DMX USB interface working on my main laptop with Win7x64 on it, however all these things work fine on an older laptop with Debian, and the older one with XP although in those cases I had to try different versions of the driver until I got one that worked.

Anyway the point is this stuff is not trivial, especially for those who's expertise is elsewhere.