How can i physically connect an image sensor (500+fps) to FPGA board ? Help!

[pollutioncontroltech]

Hi there,
I have been trying to figure out what kind of hardware/software is needed for my project. The project is getting the images with a high speed image sensor with a frame rate of 500-1000fps at a lower resolutions such as 160x120,  480x360 pixels or @VGA (any of resolution is okay). The frames will be read with FPGA SoC board and will be saved/sent to external hard drive via FPGA board.
While I was searching online I came across this project/kit (attached in pdf format) from Xilinx/Avnet. It has an Avnet FMC-IMAGEON (FMC daughter board) with VITA-2000 (Phyton-1300 C as option too) camera/image sensor with Xilinx ZC702 board. It would read the images at 60fps with 1080p resolution, and then do the other tasks ( display the images etc.)
Also I came across with this news ( link is here : https://www.invision-news.de/allgemein/no-bandwidth-bottlenecks/ ) and attached picture ( main board with 2 Basler BCON for LVDS cameras/image sensors). This embedded solution is perfect for my project.
I am basically looking for something like this, ( the attached project has lower frame rate). I know that the necessary arrangements in codes/IP’s shall be made. Other than that, I have these questions popped up in my mind:
Q1: Can this project come to life  to do 500+fps with lower resolution (160x120pixels) with single Basler BCON for LVDS camera/image sensor on FPGA board? If so what kind of interface should FPGA board has to have to be able to connect Basler BCON for LVDS camera/image sensor to (i.e. a pmod or a FMC ? )
Q2: If not, what is the bottleneck and is there a way to fix it?
Q3: Do you happen to know a similar embedded vision kits that capable of doing my project?
I am interested in Embedded Vision Kits like this, and board level cameras/image sensors but couldn’t find any similar kits that has an image sensor that has a frame rate that my project requires (500-1000fps at any resolution). If there is one you worked with/know would you send me a link and furthermore I would have some more technical question to be answered.
Regards
Sirac KAYA

[PhillyGreg]

Some thoughts,

If you look at the datasheet for the Dart module, it gives more information on the BCON connection.
It maxes out at 84Mhz, and you can send at most two pixels per clock cycle so 168M pixels/second

500 frames/sec is 168Mpixels/500=336,000pixels/frame which is 578x578 so not too bad
1000 frames/sec 168kpixels so ~400x400 frame

That's assuming you find a camera with that interface, that meets your frame rate and optical performance requirements. Looking on Basler's site, they don't make one.

I'm not too familiar with the offerings for small CMOS sensors. (My interests are in 1" format and 35mm format. Namely the CMV2000, CMV4000 and a couple DSLR sensors). Most of them seem to be moving to 4 - 12 lane sub-LVDS or mipi-csi, which can get you the frame rates you want. Any of the 7 series Xilinx has app notes on mipi-csi phy solutions, and I've had no problem with 250Mhz DDR signals on a Zynq(Artix fabric) board.

It may be easier to find a sensor that meets your performance requirements and build your own header board to mate with an FPGA board.
Forget about PMOD connectors, not going to give you the performance you want. They are for low speed signals, depending on the board design this can be pushed but I wouldn't rely on it. For most image sensors, data rate per lane isn't too high, so any good HS board to board connector would be fine. FMC is kinda a pain to deal with sometimes.

The bigger part of the problem is going to be buffering the frames and doing something with the data after you capture it.
Which FPGA SoC board are you targeting?

[pollutioncontroltech]

Thank you @PhillyGreg for the response, i appreciate it!

I think Basler does make one (link here : https://www.baslerweb.com/en/products/cameras/area-scan-cameras/dart/daa1600-60lm/ )
There is no info about max frame rate, i emailed the company and they said these image sensors are capable of doing 500+fps.

I have these options to do this job but really not capable of making the decision wisely since i am just newbie and have done small projects with FPGAs.

Option1 : Basler dart BCON for LVDS Development Kit
Link: https://www.baslerweb.com/en/embedded-vision/embedded-vision-portfolio/embedded-vision-kits/basler-dart-bcon-for-lvds-development-kit/
I am thinking this option with a different image sensor that does the job (500+fps, sensor link is given up there).

Option2: Xilinx Zynq UltraScale+ MPSoC ZCU104 Evaluation Kit
Link: https://www.xilinx.com/products/boards-and-kits/zcu104.html#hardware
I am thinking this option, because it has FMC , and has also an application with a daughter board from avnet that does 120fps with phyton 1300 c image sensor. (link is here: https://www.xilinx.com/support/documentation/application_notes/xapp794-1080p60-camera.pdf

Option3: MicroZed Embedded Vision kits
Link: https://www.avnet.com/wps/portal/silica/products/product-highlights/2016/microzed-embedded-vision-kits/
I am thinking this option because it has some application made before (none of them is at frame rate that i wanted).

Option4: Digilent Embedded Vision Bundle
Link: https://store.digilentinc.com/embedded-vision-bundle/
I am thinking this option because it has some application made before (it is not at frame rate that i wanted). Ant it has an easy interface with camera module (image sensor doesn't have high frame rates though).

So, there are these options that i have for beginning, i don't think the data rates ( we will be needing as small as we can get- pixel wise so QVGA 320x240 x 1000fps ~  77Mpixels/second . So i think the bandwidth is fine ) will be an obstacle for us.

All i am expecting is just easy plug between the camera/image sensor and the board (like option 4  : it has FMC ribbon cable to connect the image sensor on the board).

You have mentioned, you have done 250Mhz DDR signaling with Artix. I am assuming you have done it with one of CMVxxxx sensors you mentioned. My question is how did you physically connect the sensor with FPGA, what ports/headers did you used?

My main problem is actual/physical connections so far, once i find a solution then i will learn about the interface too.
After i capture the data i aim to store it to edit later on. I will be saving only short time of the data (about 1GB maybe, so 10sec.)
Saving data could be in Sd card or an external hard drive. And the data type doesn't matter so far.

What would you suggest under these circumstances ?

Thank you so much for your time !

Regards
Sirac

[asmi]

The choice of FPGA (or SoC) depends on what kind of processing is going to be done. If you only want to save short bursts of imagery, Artix will be more than enough (look at AC701 devboard), however if price is not a problem, you can go for Zynq ZC702 board and leverage Linux OS to make working with file system easier (though you will have to figure out how to write a driver for Linux for your custom logic, so it might be a wash in the end). They both have FMC connectors with sufficient number of pins broken out such that you can connect the camera.

[PhillyGreg]

My question is how did you physically connect the sensor with FPGA, what ports/headers did you used?

I build my own sensor boards, I use whichever high speed connection is available on the board.
I've used FMC connectors back on a Spartan 6 project, MicroZed uses yet another header.

The CMV2000 is connected to a MicroZed, through a custom board I designed. That camera is actually the Axiom Alpha board stack with the CMV2000 because dropping 2k on a sensor was out of my comfort zone at the time. The Axiom folks only designed a board for the CMV12000, so I designed a board and used their code/HDL as a starting point.

I've gotten to the point where existing dev boards/modules don't have the features I need. Currently bringing up my first Artix board from scratch.

Getting the signals in is pretty easy in that regard, either build a custom sensor board that has the sensor and mates to say FMC, or build an adapter board for a commercial camera.

have done small projects with FPGAs.

What boards have you used for these? If you still have that board it may be a good starting point, pick a slower sensor, work out the interfacing, build it, make it work.
Then scale up to something bigger. The faster you go the more attention to have to pay to board interconnect, SI, HDL optimization.

The choice of FPGA (or SoC) depends on what kind of processing is going to be done.

This is the part of FPGA work that can drive me nuts. I have a half dozen dev boards because of this.
Honestly before you start buying boards, and building things, work out the HDL and simulate your designs.
High speed designs can be very sensitive to pinout and place and route in the chip. If you haven't already learn how to use timing constraints effectively to make sure designs meet timing.

Keep in mind, the ZCU104 board is a steal at $895, but the chip on there is ~$3k on Digikey, so if you have any plans to package this as a device and not a breadboard project keep overall device prices in mind when looking at a board. It's straight forward to go from the 200T on the AC701 to a 35T in a custom board. But if you develop on an UltraScale and then decided you want to use a MicroZed, you could be in trouble.

After i capture the data i aim to store it to edit later on. I will be saving only short time of the data (about 1GB maybe, so 10sec.)
Saving data could be in Sd card or an external hard drive.

Being able to cache the video then flush to disk will make things much easier. That's been my only problem with the Zynq stuff is not spending a fortune on a board with transceivers to drive SATA or PCIe.
If you are comfortable writing Linux drivers, and can deal with slow writes to an SD card, or write to a network drive the Zynq can be a good place to start.

[asmi]

I've gotten to the point where existing dev boards/modules don't have the features I need. Currently bringing up my first Artix board from scratch.

How's your project coming along?

This is the part of FPGA work that can drive me nuts. I have a half dozen dev boards because of this.

This is why I'm currently working on a "module" design that has only bare minimum of components (FPGA, PMIC, QSPI Flash and DDR3), and FMC connector which exposes a ton of pins - so that I can reuse these modules and make main boards cheaper as it can have only 4 layers - and even that is only for SI reasons. I plan to use 484 pin packages since they have MGTs, plus this allows me to have two footprints for DDR3(L) chips for additional flexibility - so that I can place either one (for x16 bus) or two (for x32 bus) memory ICs without board respins ( ) It will also have all 4 MGTs exposed on FMC in case I will want to use it for SATA, or PCIE, or whatever else. Once I will be sure that the board layout works ( ) I plan to order a ton of these boards (since most of the cost will be in NRE, reorders should be relatively cheap).

Honestly before you start buying boards, and building things, work out the HDL and simulate your designs.

No kidding - I already got almost burned by this when I designed a PCIE board for A50 - only to find out that design will not fit into that chip But it's not always possible - especially if you need to interface with some kind of peripheral which doesn't have very good documentation.

Keep in mind, the ZCU104 board is a steal at $895, but the chip on there is ~$3k on Digikey, so if you have any plans to package this as a device and not a breadboard project keep overall device prices in mind when looking at a board. It's straight forward to go from the 200T on the AC701 to a 35T in a custom board. But if you develop on an UltraScale and then decided you want to use a MicroZed, you could be in trouble.

I recently discovered that lower-end Zynq MPSoCs are not that expensive - considering what they provide (4 PS-GTRs, which can be used for PCIE Gen2, SATA-III, DisplayPort 1.2a, USB 3.0, etc.), so they might be a viable option if you need these (and the horsepower of powerful ARMs of course). The real annoyance though is that these lower-end parts are available only in 0.8 mm pitch BGAs, which means breaking out DDR3/4 is going to be a lot of "fun" , and they don't have any MGTs beside ones that are in PS.

Being able to cache the video then flush to disk will make things much easier. That's been my only problem with the Zynq stuff is not spending a fortune on a board with transceivers to drive SATA or PCIe.

You can buy a board with Z012S or Z015 for reasonable $$$ (or built it) - they have less logic that Z020's (and "S" version only has a single core), but they do have a full quad of MGTs.