Heavily configured FPGAs

[AussieBruce]

Can anyone give me any information on what happens when an FPGA is ‘heavily populated’, ie. as the number of elements used approaches the total number on the device? Will the IDE be any help, eg. by providing warnings? Are there any guidelines on things like the % utilisation above which trouble might emerge? And are there design procedures that can help extend the loading? (appreciate that this last question might require an entire book – or an entire career - to answer).

[ataradov]

The design will either fit into the device  while meeting all the constraints or not. There is no other option.

It depends on the vendor and set strategy, but often tools will optimize just enough to fit the requirements, and will not try to do the best job. As utilization goes up, tools will just have to try harder and harder.

This may create a huge issue, as one day a minor change may no longer make it possible to place the design, even of there are seemingly some resources left. There may not be enough routing resources while plenty of registers left, for example.

There is no direct advance warning of that as far as I know. An indirect warning is that the design all of a sudden takes too long to place.

And the thing that becomes more important as utilization goes up is that your constraints are actually setup correctly.

The general rule of thumb we used is that no more than 80% of any resource (except for pins) must be used. After than a design is either optimized or upgraded to a higher capacity FPGA. But that is for generic logic.  Some designs for DSP were big, but very pipelined and were explicitly optimized for the FPGA architecture (down to manual placement of blocks). In that case 95+% utilization was possible with no issues.

[james_s]

The most obvious effect I've observed is that compiling (or fitting or whatever you want to call the process) takes a lot longer when the device is very full. IIRC it can take several times as long as it grinds away trying to optimize the design enough to fit.

[hamster_nz]

For higher performance designs, the Fmax will usually also fall due to routing congestion.

Also note that some FPGAs come in different marketing sizes, but they have the same physical die with different device ID fusing.

So sometimes moving up to the next size up in the same range doesn't give you more physical resources and will make zero impact on performance or build times.

[SiliconWizard]

Yep. As utilization grows, the PAR (place-and-route) execution time will tend to grow a lot. And yes, Fmax will tend to decrease as well. For a high-utilization design you may need to take steps to direct the PAR process through a number of constraints, which is never trivial.

A recent example is a "small" RISC-V SOC I've implemented on a Lattice ECP5-25 (25kLUTs). Depending on how I configure it, it takes between 60% and 85% of the total LUTs. At around 60% utilization, PAR time is about 4-5 minutes (yeah it's already a moderately complex design). At around 85%, it can take over 30 minutes.

As a rule of thumb, experience has shown me that starting with 75-80% LUTs utilization, on most FPGAs, things start to get hairy.

[Someone]

Rarely are such simple answers as above possible, "resources" can be the obvious lots/flops/ram/dsp but also fabric with various flavours of routing and interconnect restrictions, and "clock" routing.

Fmax dropping is simply one indicator that there is competition ("congestion") for optimal placement/routing. What works well in isolation/optimal conditions will be impacted by congestion one way or another. Designs need to balance their frequency targets and slack against the utilisation, either one is easy to achieve in isolation, inbetween you are walking a Pareto front. The place and route mechanisms are quite noisy so its not a smooth solution space, but relying on many repetitions to find the magic seed and close timing is a losing battle. Usually time is better spent cleaning up the system and including more accurate constraints (slacken off all the things which arent actually needing cycle to cycle timing, don't be afraid of multi cycle and asynchronous paths).

100% utilisation (of the silicon, not a synthetic resource constraint) in RAM and/or DSP is easy enough to achieve at reasonable clock rates, say 2/3rds of absolute Fmax. Rarely does that need hand placement, unlike the clocking resources! So you have the tradeoff, is it easier/better/sensible to run with a slightly lower speed and more of the resources used or leave some room for the place and route to squeeze out some extra MHz. From my experience higher utilisation usually wins out in almost every way.

[AussieBruce]

Hope all responders read this. Just my thanks (aside from pushing the button) for such a useful, and clearly authoritative, set of replies.

[AndyC_772]

In my experience, the biggest problem with a near-fully utilised FPGA isn't so much technical as commercial.

On day one, you design a board with an FPGA that's a good fit for your design, including the features that are needed with a reasonable - but not excessive - amount of free space for future development. FPGAs are expensive, of course, so you pick the device you need, not one which is arbitrarily bigger. All is well.

Over time, features get added. The FPGA utilisation increases. "Can you just add...?" takes its toll.

If the design has a long lifetime - which a good, successful one will - you end up maintaining a device that has a much higher utilisation that you might like. Changes take longer and longer, as each one pushes the device's internal Fmax below what it needs to be. Each change you need to make has unintended consequences elsewhere, as you end up having to optimise whatever logic is now running a little too slowly. The risk of bugs creeping into code that was previously tried and tested, and which should have remained completely untouched, rises exponentially.

You can't now specify a bigger device, because the new software must always run on existing hardware.

[NorthGuy]

There are often series of devices with the same pinout but different amount of resources. You can use the biggest one for development and a small-ish one for production.

[SiliconWizard]

There are often series of devices with the same pinout but different amount of resources. You can use the biggest one for development and a small-ish one for production.

Absolutely. Only when you get to the final stage of your development can you select an appropriate part.

Just keep in mind what we said above, because for FPGAs, generally speaking, it's not at all like just selecting a MCU in the same series with less memory.

If all your dev was done on a much bigger FPGA, and then you select one that has barely enough resources, you are likely to run into a series of issues, and if Fmax is a concern in your particular design, you might not even be able to meet timings in some cases. So, I'm sure this is obvious, but when selecting what you think would be the final part, always test it by at the very least running a full implementation with the final part, and check that there are no timing violations. Never rush a part selection without checking. Even if you're pressured by your management in order to lower costs.

[BrianHG]

Never rush a part selection without checking. Even if you're pressured by your management in order to lower costs.

OMG, take this advice even further if development time is a factor.  Work with a component twice as large as you need.  Demand it.  First get the design working.  Then worry about fitting your design in a smaller device.  If your dev time turns out to be a year, other FPGAs at different price points and sizes may also be available at the time.

Larger FPGA not completely full take a fraction the time to compile and test.  The difference between a 70% full FPGA and 95% full one can change compile times from 4 minutes to 14 minutes or worse if you are pushing FMAX.