Best way to get stereo camera depth data from FPGA -> RPi ??

[David1234]

I have developed a sub-$150 FPGA system that generates real-time stereo camera depth/disparity data at 30+fps, at VGA resolution, with no lag. I plan to sell this as a super cheap alternative to $20k+ outdoor, 3d LIDAR systems. The last step is to figure out the best protocol for outputting this data into a raspberry pi or beaglebone. What protocol can support 307,200, 8-bit values, 30 times per second (9,216,000, 8-bit values per second)? SPI? (This is assuming that I'm not sending an address with each depth value. If an address is needed then the math works out to 9,216,000, 19 addr bits + 8 data bits per second. Can SPI on the Pi support that?

[ataradov]

USB or Ethernet are the only realistic choices. There is no way you can make SPI work at 75 MHz reliably.

And a standard USB video would be the perfect choice. There are plenty of libraries that can take the video and do further processing on it.

[David1234]

I would love to output USB video, but remember, the master device is the FPGA. I heard that it was totally infeasible for an FPGA to somehow output USB video by itself. Do you know of a break-out board for some ASIC that will generate the required USB signals?

[ataradov]

Cypress makes generic parallel port to USB bridges. That is probably the easiest way. You just need to research what they need to stream video.

In theory it is pretty easy. You need to supply correct descriptors and then just stream the data over isochronous endpoints. Cypress chips have embedded MCU that can handle all the slow speed stuff like control endpoints.

There must also be chips specifically for video streaming. But I have not used  any, and I don't know how easy they are to get.

[NorthGuy]

I don't know if RPi can support parallel port, but if it does, 10 MHz parallel port is the easiest option.

If you go USB route - I would try FT232H. It can convert USB CDC to what they call "parallel FIFO interface" which lets you achieve 40 MBytes/s communication with FPGA, according to FTDI specs. It is much cheaper than Cypress, and also has built-in drivers for all major OSes in case you decide to drop RPi. It requires some external parts - oscillator, EEPROM etc., but nothing major. Here's the link:

https://www.ftdichip.com/Products/ICs/FT232H.htm

100BaseT Ethernet may be ok, but may be too slow, depending on how you use it. Gigabit Ethernet is definitely enough, but is more expensive. Both require PHY, magnetics, and connector, and also will consume some sizeable amount of FPGA logic.

[David1234]

The FPGA board I'm using has a FTDI FT2232HQ chip on it that I currently use. Would that be fast enough to get 30fps of 8 bit data out? If so, I then need to see if the Pi can support that level of serial input.

[ataradov]

But having that video data just available though a serial port will likely to be useless to potential customers. You are simplifying life for yourself, but potentially making a product nobody would want.

[David1234]

Right. USB would be best. There needs to be a breakout board that takes in parallel camera data, in whatever form, and outputs USB video stream. It also has to cost $50 or less. If it is several hundred dollars then that defeats the purpose.

[ataradov]

Well, 2 minute search resulted in finding FTDI FT602, which sounds like what you need exactly.

It is $10 on DigiKey.

I bet there are cheaper alternatives from China.

[David1234]

I don't know if RPi can support parallel port, but if it does, 10 MHz parallel port is the easiest option.

If you go USB route - I would try FT232H. It can convert USB CDC to what they call "parallel FIFO interface" which lets you achieve 40 MBytes/s communication with FPGA, according to FTDI specs. It is much cheaper than Cypress, and also has built-in drivers for all major OSes in case you decide to drop RPi. It requires some external parts - oscillator, EEPROM etc., but nothing major. Here's the link:

https://www.ftdichip.com/Products/ICs/FT232H.htm

100BaseT Ethernet may be ok, but may be too slow, depending on how you use it. Gigabit Ethernet is definitely enough, but is more expensive. Both require PHY, magnetics, and connector, and also will consume some sizeable amount of FPGA logic.

The board I'm using apparently only reliably outputs about 1Mbaud https://forum.digilentinc.com/topic/2682-uart-communication-control-with-cmod-a7/

[NorthGuy]

The FPGA board I'm using has a FTDI FT2232HQ chip on it that I currently use. Would that be fast enough to get 30fps of 8 bit data out? If so, I then need to see if the Pi can support that level of serial input.

They have different connection modes to FPGA. Plain UART, which you have on your board, is very slow. I don't think your board has wiring for faster options. You would need to connect an external FT232H breakout board or make your own FPGA board to test. What you need is "synchronous FIFO". Read the datasheet for details.

For RPi (or any other host) it'll look like USB CDC. Linux has support for it. You just open the device as a file and read the data.

[ogden]

Best way to get video into RPi is camera (MIPI CSI2) interface where RPi camera modules usually are connected to.

You would want to read following thread: https://www.raspberrypi.org/forums/viewtopic.php?t=151840

[ataradov]

Best way to get video into RPi is camera (MIPI CSI2)

Yes, I totally forgot that RPi has MIPI interface. This is definitely a way to go if you are targeting RPi.

[Rerouter]

If you approached it like a block transfer, rather than individually addressed you can keep your transmissions closer to your Ideal of about 9MB/s

E.g. both Ethernet and USB shine for large block transfers, so instead of addressing every data point, you just address the block for reassembly.

If you did want to go with SPI, so long as its on a board you control and the lengths are low, then you can definitely send that fast, but i can make no claim for how fast a raspberry pi can interpret it.

Seeing as it seems you are encoding it as a depth map image, could you not encode for the camera interface on a raspberry pi?

[David1234]

If you approached it like a block transfer, rather than individually addressed you can keep your transmissions closer to your Ideal of about 9MB/s

E.g. both Ethernet and USB shine for large block transfers, so instead of addressing every data point, you just address the block for reassembly.

If you did want to go with SPI, so long as its on a board you control and the lengths are low, then you can definitely send that fast, but i can make no claim for how fast a raspberry pi can interpret it.

Seeing as it seems you are encoding it as a depth map image, could you not encode for the camera interface on a raspberry pi?

Maybe. I so far haven't found the details needed for *sending* camera data over the CSI port. If I could find that data, I would then need to write a CSI entity. I'd rather do USB if that is feasible since it is more widely applicable.

[ataradov]

I'd rather do USB if that is feasible since it is more widely applicable.

Then FTDI FT602 or equivalent is the best option. Your device will essentially appear as a web camera and will be automatically usable by all software that can work with web cams.

[David1234]

I'd rather do USB if that is feasible since it is more widely applicable.

Then FTDI FT602 or equivalent is the best option. Your device will essentially appear as a web camera and will be automatically usable by all software that can work with web cams.

Yea, that would be ideal. I need to find a breakout board for the FTDI FT602 and an example project where someone has actually used one, or alternatively, really good documentation.

edit:
Found a breakout board
https://brtchip.com/product/umft602a-b/

[asmi]

I made a board with FT601, which is identical to FT602 as far as FPGA side goes, it's not complicated at all. I tested the board in a 1IN-1OUT configuration while only sending data to the PC from FPGA, and I got about 323 MBytes/s (~2.5 GBit/s) of real bandwidth (meaning driver overhead has been factored in, as I was measuring the speed at the application side). I believe FT602 runs only in a single channel mode, so you can expect somewhat higher bandwidth. If you curious to see the schematic, you can look here, on the page 4.

[Morgan127]

I would definitely use Ethernet. Then you get a stable and reliable product.
You will also have no problem to switch from BB to RPI or whatever board you choose. No drivers etc.
These board (linux) handles Ethernet very good.
On the FPGA side you use UDP with fix ip. I guess you have Ethernet on your FPGA board.
There exist Ethernet cores (at least for Xilinx) with all necessary functionality without use of embedded processor. Using one of them and  and you will be up and running in minutes.
I have been using cores from FPGA Ethernet Cores in similar projects. Free download

[asmi]

I would definitely use Ethernet. Then you get a stable and reliable product.

"Ethernet" and "reliable" do not belong to a single sentence. There are way too many corner cases.

I have been using cores from FPGA Ethernet Cores in similar projects. Free download

It's free only for non-commercial use, while the OP clearly started he wants to sell the product

[David1234]

I made a board with FT601, which is identical to FT602 as far as FPGA side goes, it's not complicated at all. I tested the board in a 1IN-1OUT configuration while only sending data to the PC from FPGA, and I got about 323 MBytes/s (~2.5 GBit/s) of real bandwidth (meaning driver overhead has been factored in, as I was measuring the speed at the application side). I believe FT602 runs only in a single channel mode, so you can expect somewhat higher bandwidth. If you curious to see the schematic, you can look here, on the page 4.

I think I found the most inexpensive, simplest thing that will result in the FPGA stereo system being recognized as a USB video device, the cypress FX2LP https://community.cypress.com/docs/DOC-9329 The FX3 is newer, but more expensive and not needed as I only plan to run VGA and USB 2.0 is good enough. The cypress products seem to have better documentation and examples vs the FTDI stuff.

[xyrtek]

I know you have your own hardware and I dont like to suggest something that doesn't address your question or that points you in a direction you probably considered already but....

Did you consider using a SoC?

Xilinx Zynq 7000 series comes to mind.

What functions are you trying to implementd on the raspberry pi? (SBC side)

If only to "display" the video then why not implement the VGA port on the fpga? Quiet simple actually.

If price is your concern check the good folks at krtkl, they managed to produce a sub $150 board based using the z7000 series.

https://www.crowdsupply.com/krtkl/snickerdoodle#products-top

It may seems hard to change course while you are well on your way with you project but the interconnect provided by a SoC can save you lots of time and bring true flexibility to you project.

Hope this help.

[ataradov]

What functions are you trying to implementd on the raspberry pi? (SBC side)

RPi is just an example. The device and its depth map output is the actual product. So that video output would be the "API".

[xyrtek]

What functions are you trying to implementd on the raspberry pi? (SBC side)

RPi is just an example. The device and its depth map output is the actual product. So that video output would be the "API".

I understood that part and thats why I included SBC side on my question.

Anyways without hijacking the thread, what the OP describes (what I sort of understood) real time, 3D lidar, price etc...sounds like a job for a SOC to me.

Not going to explain the advantages, that's an easy google search away and common sense.

[MarkR42]

Maybe there is a way to hack it into the Raspberry PI SDIO bus?

The SDIO bus is used on the Pi 3 to connect to the wifi, which avoids the bottleneck of the usb controller which affected wifi dongles on earlier Pis. This is how it gets better wifi performance.

I read somewhere that it is possible to hack other devices to the sdio, but it is mostly used for storage devices.

As you have a fpga, presumably it is possible to make it talk whatever prototcol you want, provided it is electrically compatible.