FPGA board for computer vision

[OM222O]

Hi everyone, I'm trying to build a computer vision system for robotics / embedded applications and have developed a proof of concept model which was tested on a CMOD S7.

That was for my master's project and despite proving the system works, it was not usable for any real applications due to the limited I/O and lack of logic blocks; now after graduation, I like to continue the project further and make a real system that is able to process 1080P images. I've had a look at the offerings on the Xilinx website but I'm not sure which platform to choose. Most image processing tutorials on youtube use the Zynq 7000 boards such as the Zedboard or the Pynq boards. I like the idea of interfacing the hardware with python, but that is not a must-have feature.

On the other hand, there are the Kira system on module boards (so far only Kria K26, but they advertise a "Highest Compute SOM" which I can wait for), that are advertised for being great at computer vision and A.I.

The most important requirements for me, in decreasing importance are:
1) memory bandwidth (hardware support for things like PCIe, DDR or GDDR, HBM, etc. would be ideal)
2) lots of IO (to maximize data transfer rates by using multiple memory modules, I want a minimum of a 256 bit bus)
3) floating point performance (I can compromise on this and use integer-only arithmetics, so a lot of "DSP slices" would be fine too)
4) a decent amount of block ram/cache (around 1KB should be fine)

The main operation used in my computer vision pipeline is convolution(multiply-accumulate), which is embarrassingly parallel and would benefit massively from memory bandwidth and hardware support for multiply-accumulate (DSP slices).

The board should ideally be around 500$, but I can stretch the budget to 1000$ if there are meaningful gains in terms of bandwidth or DSP slices. Which boards do you recommend?

[RoGeorge]

You may want to try something from nVidia, like Jetson Nano or alike, there are more in their AI/Vision devboards.  They are way more powerful than any Zynq.

Zynq is not aimed for vision applications, it's a general purpose small/medium FPGA with a hardcore ARM in the same chip.  Has DSPs and eventually can do some basic video processing, but nothing spectacular.

[OM222O]

You may want to try something from nVidia, like Jetson Nano or alike, there are more in their AI/Vision devboards.  They are way more powerful than any Zynq.

Zynq is not aimed for vision applications, it's a general purpose small/medium FPGA with a hardcore ARM in the same chip.  Has DSPs and eventually can do some basic video processing, but nothing spectacular.

you are entirely missing the points I mentioned. Those boards have next to 0 memory bandwidth and offer practically no expansion options for memory either. Yes, they have "powerful" GPUs that can detect objects etc, but that is NOT what I'm doing; it's not even close! The only advantage is the high FP64 compute, but without transfer bandwidth, that is useless.

[Someone]

You may want to try something from nVidia, like Jetson Nano or alike, there are more in their AI/Vision devboards.  They are way more powerful than any Zynq.

Zynq is not aimed for vision applications, it's a general purpose small/medium FPGA with a hardcore ARM in the same chip.  Has DSPs and eventually can do some basic video processing, but nothing spectacular.

you are entirely missing the points I mentioned. Those boards have next to 0 memory bandwidth and offer practically no expansion options for memory either. Yes, they have "powerful" GPUs that can detect objects etc, but that is NOT what I'm doing; it's not even close! The only advantage is the high FP64 compute, but without transfer bandwidth, that is useless.

Well the Kira does not have a wide memory bank, so thats out. Asking for floating point usually ends up a better match with GPU SoC.....

Your budget is unrealistic for the capabilities you ask for. Quickest way is likely a networking oriented FPGA board and some GigE cameras.

[RoGeorge]

you are entirely missing the points I mentioned. Those boards have next to 0 memory bandwidth and offer practically no expansion options for memory either. Yes, they have "powerful" GPUs that can detect objects etc, but that is NOT what I'm doing; it's not even close! The only advantage is the high FP64 compute, but without transfer bandwidth, that is useless.

Sorry that my crystal globe was foggy and couldn't guess what memory bandwidth you expect. 

You entirely missed the words "or alike" after Jetson Nano.  Nano is the cheapest entry, $50-$100.  At $500-$1000 you can buy a 32GB nVidia AGX (AGX has integrated 32/64GB DDR4 256bits wide 136.5GB/s, which is not close to zero).  I don't think a Zynq can do better.

Maybe a Virtex might go faster than an AGX ("The Xilinx DDR4 controller is high performance 2667Mbps in UItraScale+"), but that would be way out of your budget.

[BrianHG]

Doesn't the Jetson Nano have PCIE M.2 Key E port?
You should be able to DMA the onboard system mem to that port.

Though, a GPU card tied to your host CPU of choice might be a cheap but non-guaranteed stock method.
Any quality budget GPU with a good 256bit or larger GDDR6 ram bus should run circles around any price competitive FPGA operating at the same memory bandwidth scale.

[rstofer]

I don't know how to interpret the video numbers but here they are for the Nano
https://developer.nvidia.com/embedded/jetson-nano

The fact that it is also a full-blown Ubuntu system makes the board really easy to work with.

[Someone]

Sorry that my crystal globe was foggy and couldn't guess what memory bandwidth you expect. 

You entirely missed the words "or alike" after Jetson Nano.  Nano is the cheapest entry, $50-$100.  At $500-$1000 you can buy a 32GB nVidia AGX (AGX has integrated 32/64GB DDR4 256bits wide 136.5GB/s, which is not close to zero).  I don't think a Zynq can do better.

Maybe a Virtex might go faster than an AGX ("The Xilinx DDR4 controller is high performance 2667Mbps in UItraScale+"), but that would be way out of your budget.

Yes, budget is the killer. The OP would likely get much more compute per $$$ for their application (https://www.eevblog.com/forum/fpga/image-processing-accelerator/) from something high volume like a GPU or similar.

... BTW the whole point of an FPGA is ability to scale memory to your needs, there are 192bit wide dev boards:
https://www.iwavesystems.com/product/zu19-zu17-zu11-zynq-ultrascale-mpsocsom/
and the underlying Zynq devices support wider memory if needed (have worked on systems with wider busses) but are lower volume so cost is always a pressure. But its silly to constrain to zynq unless the tight integration with the ARM cores is important to the system, HBM on FPGA = crazy memory throughput

[miken]

It would be fun to get a SQRL FK33, which has a VU33P with HBM. Normally that kind of platform would be unaffordable, but thanks to crypto miners subsidizing the hardware, it's curiously within reach.

A GPU would be the obvious platform, but where's the fun in that? 

[OM222O]

HBM seems to be way too expensive and unobtainable I'm honestly not sure if I'm capable of wiring my own memory controller either, but if DDR4 /  DDR5 are the reasonable options, I don't mind giving it a shot. Also please keep the suggestions to FPGAs as I'm not interested in a GPU (the entire project has to be done from scratch rather than scaling up what I have with faster memory and more compute).

If these FPGAs are really that bad in terms of memory bandwidth, then why do all high-speed cameras use an FPGA to capture footage? I've never seen one that uses a GPU  If you actually read the first post, bandwidth is all I care about now (compute and cache are 4th and 5th priority). With that in mind, what would be a decent FPGA board in the 500 to 1000$ range?

Edit: I forgot to mention that all oscilloscopes also use an FPGA to store the ADC values in memory. They are clearly the most logical option if you want to get high memory bandwidth.

[miken]

Well, how much do high-speed cameras cost? Or high-end oscilloscopes? 

FPGA + fast wide memory is a relatively rare application so it's going to be more expensive. Most cheap(er) dev boards that have memory will have a 16 bit interface. If you want 256, consider that that's 4 DIMMs! You'd want a big Kintex or Virtex for that kind of project.

Which is not cheap, of course. You're not going to find a new dev board with those specs within your budget. If you want high performance at a low price you have to scrounge on eBay and do a lot of legwork yourself.

[OM222O]

Well, how much do high-speed cameras cost? Or high-end oscilloscopes? 

FPGA + fast wide memory is a relatively rare application so it's going to be more expensive. Most cheap(er) dev boards that have memory will have a 16 bit interface. If you want 256, consider that that's 4 DIMMs! You'd want a big Kintex or Virtex for that kind of project.

Which is not cheap, of course. You're not going to find a new dev board with those specs within your budget. If you want high performance at a low price you have to scrounge on eBay and do a lot of legwork yourself.

Kintex dev boards on the official Xilinx website are 1200$. entry oscilloscopes and high-speed cameras are also extremely common in the 500 to 1000$ range so I'm not sure how they could afford some million-dollar FPGAs while making a profit. I'm honestly not looking for some outlandish part, just want to make the most out of the money I have. Even the low-end spartan 7 boards come with 500 IO pins, but they're fairly slow (500MHz clock). Artix and Zynq seem to be step-ups from spartan but I'm not sure about the Kira SOMs since they're heavily advertised for image processing and A.I. both of which are farily bandwidth-heavy tasks.

[miken]

And what kind of memory interface does the Kintex board have for $1200?

You need to understand that your desired specs are fairly high-end as far as FPGAs go.

[Someone]

And what kind of memory interface does the Kintex board have for $1200?

You need to understand that your desired specs are fairly high-end as far as FPGAs go.

Its clear once again the OP has no intention of quantifying anything beyond their intense area of focus (memory bandwidth, a design goal itself unsupported by any engineering) so cannot make any engineering tradeoffs or a value proposition. They also have zero concept of the link between volume and price, there are off the shelf products with multiple Kintex chips selling retail for a fraction of the cost of a single part or the matching dev board while having also more memory devices and bandwidth.

There is some link between the available bandwidth and the amount of compute/DSP that could be utilised (or flipping the language around would be required) for the application. But instead those dimensions are ignored at the pursuit of most memory bandwidth for fixed $ amount.

You said it well:

Which is not cheap, of course. You're not going to find a new dev board with those specs within your budget. If you want high performance at a low price you have to scrounge on eBay and do a lot of legwork yourself.

Why would we care about value engineering and sourcing a one off purchase for the OP 

[nctnico]

Doesn't the Jetson Nano have PCIE M.2 Key E port?
You should be able to DMA the onboard system mem to that port.

Yep. And the Jetson TX2 can cluster up to 4 PCIe lanes with 5GT/s allowing up to 2GB/s. That is more than enough to transfer a 1080P video stream. Add a Coral AI module to the system (PCIe or USB) and you have a crapload of processing power.

[OM222O]

And what kind of memory interface does the Kintex board have for $1200?

You need to understand that your desired specs are fairly high-end as far as FPGAs go.

Its clear once again the OP has no intention of quantifying anything beyond their intense area of focus (memory bandwidth, a design goal itself unsupported by any engineering) so cannot make any engineering tradeoffs or a value proposition. They also have zero concept of the link between volume and price, there are off the shelf products with multiple Kintex chips selling retail for a fraction of the cost of a single part or the matching dev board while having also more memory devices and bandwidth.

There is some link between the available bandwidth and the amount of compute/DSP that could be utilised (or flipping the language around would be required) for the application. But instead those dimensions are ignored at the pursuit of most memory bandwidth for fixed $ amount.

You said it well:

Which is not cheap, of course. You're not going to find a new dev board with those specs within your budget. If you want high performance at a low price you have to scrounge on eBay and do a lot of legwork yourself.

Why would we care about value engineering and sourcing a one off purchase for the OP 

based on a design goal unsupported by any engineering? I've ran the software version on at least 100 different computers now and regardless of the compute power and number of cores available etc, the only difference in the end is directly proprotional to memory bandwidth.

These are numbers from running it on a system with a 5950x and DDR4 3800 @ CL16:




and these are the numbers from a much less powerful 6700HQ and DDR4 2133 @ CL15 (scroll to the end):
https://github.com/2332575Y/Retina-V3/blob/main/Untitled.ipynb

both systems only scale upto 2 cores (4 threads because of hyper threading) and the 5950x system appears to be roughly 2x faster, because it has 2x the memory bandwidth. This was just an example of systems that have massively different compute power which ends up being irrelevant due to the memory bandwidth. If you want, I'll be happy to share the excel sheet for all of the different systems I tested it on and again, by memory bandwidth alone you could always guess the absolute performance, regardless of the CPU used. At this point I'll just go with one of the Zynq boards as no one seems to give advice beyond a "lol, good luck with that".

[Someone]

based on a design goal unsupported by any engineering? I've ran the software version on at least 100 different computers

Yes, but you want to implement it on an FPGA....

which can be simulated and tested without having the actual hardware, to find out what the limitations/performance is.

You are stuck with this idea that optimisation/performance on CPUs is somehow correlated to FPGAs. News flash, it can be radically disconnected! The algorithm that makes sense and is most efficient on a CPU is not always the same as the algorithm that is most efficient on an FPGA.

But do go ahead and buy an FPGA without knowing if its actually going to fit to your end application. Infinite memory bandwidth with a single LUT ? useless, somewhere along that continuum is a real system.

[rstofer]

This is the output of a sample simulation from NVIDIA for the Jetson Nano with 128 CUDA cores.  Imagine what my high performance laptop RTX3070 (laptop) video card with over 5000 CUDA cores could do!  I know I should install the software on my laptop but I probably won't.

Code: [Select]

Executing Black-Scholes GPU kernel (512 iterations)...
Options count             : 8000000     
BlackScholesGPU() time    : 4.847039 msec
Effective memory bandwidth: 16.504921 GB/s
Gigaoptions per second    : 1.650492     

BlackScholes, Throughput = 1.6505 GOptions/s, Time = 0.00485 s, Size = 8000000 options, NumDevsUsed = 1, Workgroup = 128


I guess if I were into options trading this would make more sense.  Still, the numbers seem pretty high.

16 GB/s of effective memory bandwidth seems pretty high.  It's going to take a heck of an FPGA to outrun my laptop video card.  I can't even imagine instantiating 5000 instances of a CUDA component and I'm pretty sure memory bandwidth up in the GB range will be a challenge.

[nctnico]

And what kind of memory interface does the Kintex board have for $1200?

You need to understand that your desired specs are fairly high-end as far as FPGAs go.

Its clear once again the OP has no intention of quantifying anything beyond their intense area of focus (memory bandwidth, a design goal itself unsupported by any engineering) so cannot make any engineering tradeoffs or a value proposition. They also have zero concept of the link between volume and price, there are off the shelf products with multiple Kintex chips selling retail for a fraction of the cost of a single part or the matching dev board while having also more memory devices and bandwidth.

There is some link between the available bandwidth and the amount of compute/DSP that could be utilised (or flipping the language around would be required) for the application. But instead those dimensions are ignored at the pursuit of most memory bandwidth for fixed $ amount.

You said it well:

Which is not cheap, of course. You're not going to find a new dev board with those specs within your budget. If you want high performance at a low price you have to scrounge on eBay and do a lot of legwork yourself.

Why would we care about value engineering and sourcing a one off purchase for the OP 

based on a design goal unsupported by any engineering? I've ran the software version on at least 100 different computers now and regardless of the compute power and number of cores available etc, the only difference in the end is directly proprotional to memory bandwidth.

Which is more likely an indicator that your software implementation is causing a massive amount of L1/L2 cache misses on the system you are running it on.

Forget about looking for an FPGA right now and try to really understand your algorithm first. Convert it to C++ first so you have no Python cruft in between that moves data around in an inefficient way. Use a profiler to see where the bottlenecks are. Secondly, convert the algorithm to OpenCL so you can run it on a GPU and see what the performance is when running it on a massively parallel system.

[miken]

If the point is just to futz around on FPGA, then whatever, go nuts. If the point is to run it on an FPGA while making use of the platform's strengths relative to CPU/GPU, that's a much taller order. It requires understanding how to map the algorithm onto HW resources. An FPGA is _not_ a better CPU... it is much slower at raw calculations so you only come out ahead if you can make up the difference via parallelism and pipelining.

Now I'm curious what OP had running on the CMOD S7... you can't fit much in a XC7S25. On a mid-range FPGA there's a lot more BRAM so you could probably get things done more efficiently, even without a huge pipe to DRAM. Of course, it could also turn out that the algo is not suited to FPGA at all.

[OM222O]

Which is more likely an indicator that your software implementation is causing a massive amount of L1/L2 cache misses on the system you are running it on.

Forget about looking for an FPGA right now and try to really understand your algorithm first. Convert it to C++ first so you have no Python cruft in between that moves data around in an inefficient way. Use a profiler to see where the bottlenecks are. Secondly, convert the algorithm to OpenCL so you can run it on a GPU and see what the performance is when running it on a massively parallel system.

That is exactly correct (the fact that FP32 and FP64 perform the same should be an even more obvious sign that cache is not used), for a good reason too: no coefficient is used more than once in a sampling step. This is performing convolution with a set of "random" pre-defined kernels with also "random" strides, so the cache is pretty useless. The code IS in C++, just wrapped with python for ease of use. I have tried a native C equivalent and the results were basically the same. If I want to use a GPU, OpenCL is the worst option, I'll use Cuda instead, but a GPU will always be less efficient than a properly designed ASIC with an FPGA. the point everyone seems to be missing is the same repeated words: "memory bandwidth". For this application that is the single most important factor, but what do I know! I just wrote the code 

If the point is just to futz around on FPGA, then whatever, go nuts. If the point is to run it on an FPGA while making use of the platform's strengths relative to CPU/GPU, that's a much taller order. It requires understanding how to map the algorithm onto HW resources. An FPGA is _not_ a better CPU... it is much slower at raw calculations so you only come out ahead if you can make up the difference via parallelism and pipelining.

Now I'm curious what OP had running on the CMOD S7... you can't fit much in a XC7S25. On a mid-range FPGA there's a lot more BRAM so you could probably get things done more efficiently, even without a huge pipe to DRAM. Of course, it could also turn out that the algo is not suited to FPGA at all.

On the CMOD S7 I used 4 Quad SPI NOR flash modules as a 16 bit ROM to read the coefficients from and the pixel value fetched from the image was always a fixed value of 1. It did successfully run at 500MHz and transferred data between the ROM and FIFOs (ROM ran at 133MHz so again memory bandwidth was the bottleneck). If you want to check that topic here are some links:
https://www.eevblog.com/forum/projects/daisy-chaining-nor-flash-modules-to-create-a-high-bandwidth-rom/
https://www.eevblog.com/forum/fpga/issues-with-synthesis-and-implementation-of-quad-spi-flash-controller/

[miken]

Somehow I missed this thread when it was happening: https://www.eevblog.com/forum/fpga/image-processing-accelerator/
Or I blocked it from my memory. I guess you guys have been over this territory before!

If you need to MAC over an entire frame, given the streaming nature of image sensors, wouldn't you just need to match the sensor rate for your coeffs? A single 16-bit DDR3 ought to be able to handle that...

[nctnico]

It would be nice to have a very clear explaination of the OP of what the actual goal is of the algorithm (without focussing on how it is implemented). Which image processing steps are used? Why can't is be optimised to make better use of cache memory? Even the other thread doesn't make that clear at all. Having multiple threads for the same subject / question doesn't help either. Hence, nobody understands the problem in a way to give good advise.

An FPGA with an extreme memory bandwidth that exceeds a modern x86 CPU or GPU is going to be an expensive boutique item so it is probably more fruitfull to optimise the algorithm. For example: split the image in small chunks that do fit inside the cache memory.

If memory bandwidth turns out to be a problem, a more cost effective route could be to parallel more devices. Like having a bunch of Rpi compute or Jetson Nano modules on a board that perform the task in parallel. Such a system is hard to beat where it comes to memory throughput and it is scalable as well.

[laugensalm]

Not clear what you're aiming at, but I've obtained the Lattice VIP stereo vision development kit including a FX3/GigE interface card for <500 USD. So far the best FPGA based vision system I've got my hands on (tried a few from the Altera/Xilinx side as well). For prototyping custom algorithms, there's also some nice material hidden here and there.
However I'd strongly recommend to determine upfront if DDRx memory I/O is required. If yes, you might get indeed luckier with the nvidia boards mentioned above, as memory interfacing gets more complex than the vision algos themselves, likewise Zynq platforms introduce a shoeload of unnecessary overhead that are way easier solved with a drop-in RISC-V with smart hardware extensions. Depends much on whether you (want to get to) know your FPGA or rather opt for HLS 'academic' software-minded approaches.
On the downside are the costs for the Lattice tools (that are free for one year), but for learning purposes, setups with yosys will also get you up to a certain level if you write efficient verilog. Not recommended for industrial purposes though.