Cheap in-system FGPA programming

[Harvs]

I'm using a small FPGA tied to an A53 board running Linux via SPI to effectively glue a bunch of simultaneous sampling ADCs on at a decent data rate.

At the moment prototyping has been done with a machXO3, but I'll probably switch to something that's easily available in a smaller easily soldered package (e.g. the ICE40 have some nice QFN parts.)

But regardless, I want to have the capability for the A53 to do reprogramming of the device, so we can do over the network updates of all the firmware without physical access.  Looking around everyone is using FT2232h as the JTAG programmer, which will easily double cost and board area of using the FPGA.

So I was thinking of using a SPI flash and reprogramming that with the A53 while holding the FPGA in reset or something similar.

Has anyone done anything like this before?  I figure there has to be a good way of doing this, seems like it should be a normal thing to do these days.

Thanks

[Harvs]

Thanks, I obviously missed that TN.  I didn't realise the ICE40 could be an SPI slave, which makes the whole thing a lot easier. 

In face looks like I should be able to use the pins to complete configuration, then swap over to the main SPI bus afterward, which would be very ideal...

[mikeselectricstuff]

Wouldn't it be simpler to just soft-load the FPGA from the Linux system, and not use the SPI flash at all.

[Harvs]

Wouldn't it be simpler to just soft-load the FPGA from the Linux system, and not use the SPI flash at all.

Yep that's exactly what I was thinking with it being a slave.  I figured it would be master only, which would be somewhat more complex to deal with.  But being slave it looks like it should be easy to load, then use the same bus for the main 'application'.

[mikeselectricstuff]

Wouldn't it be simpler to just soft-load the FPGA from the Linux system, and not use the SPI flash at all.

Yep that's exactly what I was thinking with it being a slave.  I figured it would be master only, which would be somewhat more complex to deal with.  But being slave it looks like it should be easy to load, then use the same bus for the main 'application'.

It also means you don't have to worry about the main app and FPGA having different versions as the app effectively  carries the FPGA code with it.

[OwO]

It doesn't matter if the FPGA is programmed via SPI or JTAG, you can always bitbang the protocol on some GPIOs. In the case of JTAG the tools can usually output a .svf file which contain commands that you can parse and replay (by bitbanging).

[aandrew]

This is exactly what I'm doing with an ICE40 design I'm using. There was a bit of a trick to it (CS# must be *low* when FPGA comes out of reset so it knows it's a slave, and you must also toggle the clock a bunch of times after sending the image so that the FPGA exits configuration mode, but the trickiest part was just figuring out how to get the .bit file in the appropriate format:

under [tool options], make sure BitmapDisableHeader=yes is set, and then convert the .bin file to something like a C header that you can easily access from your program:

xxd -i foo_implmnt/sbt/outputs/bitmap/foo_bitmap.bin /tmp/fpga_image.c

[aandrew]

It doesn't matter if the FPGA is programmed via SPI or JTAG, you can always bitbang the protocol on some GPIOs. In the case of JTAG the tools can usually output a .svf file which contain commands that you can parse and replay (by bitbanging).

You can always take the bruteforce approach and JTAG it like this, but many FPGAs let you send an image in easier ways.

[technix]

AFAIK all recent-ish SRAM FPGAs from Xilinx and Intel can SPI boot. And it can be done using spidev driver and a shell script doing "cat /usr/lib/fpga/bitstream.bin > /dev/spidev0.0" at its heart. This means you can skip FPGA configuration memory and instead integrate the FPGA boot process into the Linux kernel boot process, and system updates to the CPU-side software can also update FPGA by packing in a new bitstream file. If you have wired the master reset pin of the FPGA to the CPU, you can even hot reinitialize the FPGA.

[SiliconWizard]

It doesn't matter if the FPGA is programmed via SPI or JTAG, you can always bitbang the protocol on some GPIOs. In the case of JTAG the tools can usually output a .svf file which contain commands that you can parse and replay (by bitbanging).

Yes. You can either emulate an SPI flash chip, program it as an SPI slave or use JTAG. Many options.
The SPI slave approach is probably a bit easier? But yes from SVF files it's still relatively easy to use JTAG.

[NorthGuy]

It's not just SPI. You need to manipulate/test other pins to enter the programming mode and verify the results.

With JTAG you reset/check state through JTAG commands, and also can do other things - such as read device id or temperature/voltage sensors. You can also access JTAG from your design, so you can establish communications between FPGA and the other end.

Either way you can use configuration flash, but with SPI you need to manipulate mode pins to switch between SPI boots and flash boots. With JTAG you just do nothing and it configures from flash. If you wish you can program the configuration flash through JTAG. You probably can do it through SPI too, but it requires more work.

Thus, I would go with JTAG over SPI.

[Bassman59]

Has anyone done anything like this before?  I figure there has to be a good way of doing this, seems like it should be a normal thing to do these days.

Once upon a time I hung an SPI flash part off of an SiLabs 8051, and the 8051 read from that flash and bit-banged each byte to a Virtex-4's configuration port. The SPI flash had enough memory for two configurations, and one or the other was loaded when necessary if the user changed a system setting that demanded a different FPGA config.

The whole system connected to the host computer and a standard UART COM port was used to send commands to and get status from that 8051. One could use XMODEM to upload new FPGA configurations.

What I'd do now depends on the system design and what interfaces are available.

[technix]

It's not just SPI. You need to manipulate/test other pins to enter the programming mode and verify the results.

With JTAG you reset/check state through JTAG commands, and also can do other things - such as read device id or temperature/voltage sensors. You can also access JTAG from your design, so you can establish communications between FPGA and the other end.

Either way you can use configuration flash, but with SPI you need to manipulate mode pins to switch between SPI boots and flash boots. With JTAG you just do nothing and it configures from flash. If you wish you can program the configuration flash through JTAG. You probably can do it through SPI too, but it requires more work.

Thus, I would go with JTAG over SPI.

JTAG requires bit banging, thus the maximum speed is fairly restricted. If you are booting something like a XC6SLX9 it is tolerable, one step up to XC6SLX100 you would be asking for trouble using any bit banged interface, let alone beasts like XC7VX1140T.

For chips like XC6SLX100 I need hardware SPI for the speed. It is almost impossible to bit bang JTAG from Linux kernel at a bus speed of a few megahertz, but piping a mmap'd file down SPI hardware using DMA can achieve speeds no less than tens of megabits per second or above depending on SoC used.

As of that beast of a chip known as XC7VX1140T, I would have to use at least 16-bit SelectMAP attached to the SoC's EBI SRAM interface and dump the mmap'd bitstream in using memcpy. That can give hundreds of megabits per second.

[NorthGuy]

JTAG requires bit banging, thus the maximum speed is fairly restricted. If you are booting something like a XC6SLX9 it is tolerable, one step up to XC6SLX100 you would be asking for trouble using any bit banged interface, let alone beasts like XC7VX1140T.

You bit-bang it to the point where you're about to output the bitstream. Then, it's the same as SPI during bitstream loading as TMS doesn't need any manipulations. Thus, you can use the same method as with SPI. Then you go back to
bit-banging to verify the status. So, it is not any slower than SPI. Other times, you can bit-bang it to do other neat things.

FTDI FT232H can do JTAG at 30 MHz. After you sold your house to buy XC7VX1140T, would you have $5 left for an FTDI chip?

[Morgan127]

For in field updates we use these Ethernet Cores. These works for Spartan 6 and other Xilinx FPGAs. However that requires that you got an Ethernet port on your board and that your FPGA boots in master SPI mode.

[SiliconWizard]

For in field updates we use these Ethernet Cores. These works for Spartan 6 and other Xilinx FPGAs. However that requires that you got an Ethernet port on your board and that your FPGA boots in master SPI mode.

Well, as I understood it, the OP wants to be able to reprogram the FPGA via the main processor, which already has a network connection (ethernet or WIFI?), so adding a second network port dedicated to reprogramming would be cumbersome here and more costly anyway.

Also I don't know if your field update system implements full TCP/IP connections or just raw point-to-point ethernet, in which case it would be a lot less flexible than the OP's solution especially if they have to deal with several devices in the same network.

[technix]

You bit-bang it to the point where you're about to output the bitstream. Then, it's the same as SPI during bitstream loading as TMS doesn't need any manipulations. Thus, you can use the same method as with SPI. Then you go back to
bit-banging to verify the status. So, it is not any slower than SPI. Other times, you can bit-bang it to do other neat things.

There would be pinmux conflicts all over the place if I am loading it from Linux, as it really don't want to permit dynamic pin reallocating. Or I have to use both SPI and GPIO pins, and add an external mux chip. SPI-only mode alleviates all of that, and it still allows verification depending on exact implementation.

Additional benefit of native SPI is that after the FPGA booted the logic programmed in there can take over the SPI pins allowing the same set of pins being reused for actual application.

FTDI FT232H can do JTAG at 30 MHz. After you sold your house to buy XC7VX1140T, would you have $5 left for an FTDI chip?

I am just using that chip as an example of a "crazy big SRAM FPGA that has a ginormous bitstream file" that can require ages to be sent over either JTAG or SPI. Also the EMIF-to-SelectMAP link also allow both rapid FPGA configuration (using memcpy here as the FPGA is mapped directly to SoC memory space) and communication between SoC and FPGA post configuration with the same set of pins.

[NorthGuy]

There would be pinmux conflicts all over the place if I am loading it from Linux, as it really don't want to permit dynamic pin reallocating. Or I have to use both SPI and GPIO pins, and add an external mux chip.

Yes, people use these Linux systems thinking it's a great power (or great progress), then they cannot do a simple thing which is a snap on PIC16. Go figure.

FTDI FT232H can do JTAG at 30 MHz. After you sold your house to buy XC7VX1140T, would you have $5 left for an FTDI chip?

I am just using that chip as an example of a "crazy big SRAM FPGA that has a ginormous bitstream file" that can require ages to be sent over either JTAG or SPI.

Ages? It is not that bad. 385 Mbit bitstream. About 12 sec at 30 MHz. Compared to all that time you spend to generate a bitstream for it in Vivado ...

[technix]

Yes, people use these Linux systems thinking it's a great power (or great progress), then they cannot do a simple thing which is a snap on PIC16. Go figure.

Bare metal programming and embedded Linux each have their benefits and weaknesses. For example AFAIK the network, security and GUI on linux is much more mature and proven than that for embedded systems, as Linux code for those components are derived from code intended for servers (network and security) and Android (GUI.) Being a standard UNIX system it allows direct code reuse from other platforms.

Ages? It is not that bad. 385 Mbit bitstream. About 12 sec at 30 MHz. Compared to all that time you spend to generate a bitstream for it in Vivado ...

I want the final system to boot in a few seconds, not minutes. The bitstream file is generated once per revision on a high powered workstation, but it is loaded a lot more frequently using a relatively low speed processor.

[NorthGuy]

I want the final system to boot in a few seconds, not minutes. The bitstream file is generated once per revision on a high powered workstation, but it is loaded a lot more frequently using a relatively low speed processor.

My Artixes load in under a second through JTAG with FT232H. A good portion of that time is loading of the FTDI DLL. If you use your own chip, you thereby eliminate DLL bloat and can use 66 MHz signaling. Then it'll be around 300 ms, even faster if you compress the bitstream.

[technix]

My Artixes load in under a second through JTAG with FT232H. A good portion of that time is loading of the FTDI DLL. If you use your own chip, you thereby eliminate DLL bloat and can use 66 MHz signaling. Then it'll be around 300 ms, even faster if you compress the bitstream.

Combine that with the SPI/SelectMAP signal reusing, that is why I prefer SPI over bit bang JTAG.

[legacy]

What does the FPGA need to implement?

Cannot you have an SPI-flash multiplexed to both FPGA and SoC?

This way when the mux connects the flash to the SoC it can program the flash once and for all (even with a slow method), while in the default configuration the mux connects the flash to the FPGA so it can bootstrap in parallel to the SoC.

Why do you need to load the bitstream at every bootstrap?

[SiliconWizard]

Yeah, something has not been completely cleared up yet IMO.

Is the OP going to use an FPGA with embedded Flash, NVM or not? The MachXO3 can come in different versions I think, some with Flash, some with NVM that can be reprogrammed just a couple of times. The iCE40 line, unless I've missed something, doesn't have any kind of Flash or NVM and thus requires to be configured upon every power-on. The OP may also select a Xilinx part or whatever, which would be in the same case.

So if the FPGA has embedded Flash, reprogramming it at every power-on doesn't really seem to make sense. Reprogramming should only be required for updates. But if it doesn't, you'll have to program (configure) it every time. Not exactly the same use cases already.

In any case, if using a "biggish" FPGA (a large bitstream), and you want to reconfigure it at each power-on, emulating an external SPI flash chip would probably yield the fastest "boot" time. If it's a small bitstream, JTAG would be perfectly fine, even at a slowish rate.

To get the best of both worlds, I may suggest implementing both. Just a few more I/Os needed. Emulate an SPI flash IC, but also implement JTAG. With the added JTAG, you have additional means of debugging, re-reading the content, etc. Problem with emulating an SPI flash chip at an high clock freq is that it may be impossible to do from a typical CPU (much easier to do with an MCU with an SPI slave peripheral). If the CPU can do SPI slave, and you run Linux on it, and you have proper drivers available, that would be an option. You could also use an intermediate SPI Flash or RAM chip. Simply bit-banging JTAG from the CPU itself could be much slower than what you'd get with an MCU. Using an intermediate FTDI chip is of course an option, but the OP rejected it.

There are of course other options as seen above, so that's just a suggestion.

[technix]

In any case, if using a "biggish" FPGA (a large bitstream), and you want to reconfigure it at each power-on, emulating an external SPI flash chip would probably yield the fastest "boot" time. If it's a small bitstream, JTAG would be perfectly fine, even at a slowish rate.

Flash emulation may not be possible given certain SoC used. For example BCM2837 of Raspberry Pi 3 lacked either SPI slave or I2C slave interface. If you have a SoC it is likely better to put the FPGA in SPI slave mode instead of SPI master mode for the sake of compatibility.

[SiliconWizard]

In any case, if using a "biggish" FPGA (a large bitstream), and you want to reconfigure it at each power-on, emulating an external SPI flash chip would probably yield the fastest "boot" time. If it's a small bitstream, JTAG would be perfectly fine, even at a slowish rate.

Flash emulation may not be possible given certain SoC used. For example BCM2837 of Raspberry Pi 3 lacked either SPI slave or I2C slave interface. If you have a SoC it is likely better to put the FPGA in SPI slave mode instead of SPI master mode for the sake of compatibility.

Sure, hence all the "ifs" in my last paragraph. I just think it would be the faster option if it's possible. One additional thing is that yes, MachXO2/3 and iCE40 support SPI slave configuration, but (correct me if I'm wrong) I don't think the Xilinx parts do. So it would also all depend on the FPGA you select.

One last remark and, really obvious one, but we don't know what the FPGA is going to be used for. In the system, it may be required to be fully operational after a short while and BEFORE the SoC would have ended its booting and gotten a chance to upload the bitstream to the FPGA. Of course in that case, unless you're using a very lightweight OS which boots blazingly fast or you are using it bare metal, using an FPGA with embedded Flash (or using an external Flash IC) would be the only viable option. The SoC could then just update the Flash content if needed.

[asmi]

Xilinx provides an appnote (XAPP583) with sample code on how to boot their FPGAs using MCU. Practically if bitstream doesn't require often updates, I'd just implement it the standard way - put it on QSPI flash, and whenever I needed to update it - I would program FPGA to become a bridge of sorts between AP and FPGA, and write it to flash indirectly via said FPGA. Kinda like what Xilinx and Digilent programmers are doing when programming flash from within Vivado.

[technix]

Sure, hence all the "ifs" in my last paragraph. I just think it would be the faster option if it's possible. One additional thing is that yes, MachXO2/3 and iCE40 support SPI slave configuration, but (correct me if I'm wrong) I don't think the Xilinx parts do. So it would also all depend on the FPGA you select.

Xilinx and Intel parts all supports SPI slave mode, at least recent ones like Spartan 6 and above.

One last remark and, really obvious one, but we don't know what the FPGA is going to be used for. In the system, it may be required to be fully operational after a short while and BEFORE the SoC would have ended its booting and gotten a chance to upload the bitstream to the FPGA. Of course in that case, unless you're using a very lightweight OS which boots blazingly fast or you are using it bare metal, using an FPGA with embedded Flash (or using an external Flash IC) would be the only viable option. The SoC could then just update the Flash content if needed.

If the SoC intends to run Linux, the FPGA can be initialized within u-boot. u-boot being a bootloader means it does not require much to start up, and it also means that FPGA would be reliably booted up before kernel loads.

[legacy]

I have here a Blackfin board made by NOVA System, it also has a tiny FPGA on it. The DSP loads a customized version of u-boot, which is able to (upload a file from the network in order to) "program" the flash from which the FPGA does boot. Once programmed, it sends a reset signal to both chips, and the DSP and FPGA bootstrap singularly and in parallel.

p.s.
Note there are three SPI-flash chips on the PCB. The first is used for the bootstrapping of the DSP, the second is used for the booting of the FPGA, and the third is used as "fail-safe".

SPI-flash0: DSP firmware, u-boot + RTOS
SPI-flash1: FPGA bitstream
SPI-flash2: DSP fail-safe, u-boot stable version
(default configuration)

u-boot is able to program both the DSP's flash and the FPGA's flash, and there is a "fail-safe" procedure, so you accidentally reprograming the DSP's flash with the wrong file, you can always set a jumper to swap the first and the third SPI-flash.

'SPI-flash0: DSP fail-safe, u-boot stable version
SPI-flash1: FPGA's bitstream
'SPI-flash2: DSP firmware, u-boot + RTOS <----------- garbage, it needs to be reprogrammed!
(with the "fail-safe juper" set)

edit:
typos

[Harvs]

Hi all,

Obviously I missed a bunch of discussion here after the thread was dormant for a while.

The application is pretty straightforward.  The FPGA is just coordinating time sync'd sampling from multiple ADCs, packetizing it, then streaming that into the SoC via SPI and DMA on the SoC.  No fast on-time requirements.

The ICE40 fills the design requirements very well, and given it can have the bitstream loaded over SPI with the FPGA as a slave, I can't see any better way of doing it.  As Mike pointed out, keeping the SoC software and FPGA bitstream together makes update management a breese.

Interesting discussion though.

[technix]

As Mike pointed out, keeping the SoC software and FPGA bitstream together makes update management a breese.

If you wired up the FPGA allowing the SoC to reset and reinitialize it using a GPIO pin, you can even hot-patch the bitstream.

A UNIX-like OS like Linux allows software be updated without the whole system being rebooted as long as the kernel and maybe libc is not touched (with recent kernels libc hot patching is possible, even kernel hot patching is doable to some extent.) With that GPIO pin your software can reset the FPGA upon launch, upload the bitstream and do its thing. Since the FPGA is reset whenever the software launches, the software update can immediately take effect without a full system reboot whenever your software relaunches. If you decoupled the user interface from the software that actually interfaces the FPGA, updates can be a seamless experience.

[SiliconWizard]

Sure, hence all the "ifs" in my last paragraph. I just think it would be the faster option if it's possible. One additional thing is that yes, MachXO2/3 and iCE40 support SPI slave configuration, but (correct me if I'm wrong) I don't think the Xilinx parts do. So it would also all depend on the FPGA you select.

Xilinx and Intel parts all supports SPI slave mode, at least recent ones like Spartan 6 and above.

OK thanks for the info. Never had to use this on Xilinx parts and was not sure it was even supported. That's good to know!

One last remark and, really obvious one, but we don't know what the FPGA is going to be used for. In the system, it may be required to be fully operational after a short while and BEFORE the SoC would have ended its booting and gotten a chance to upload the bitstream to the FPGA. Of course in that case, unless you're using a very lightweight OS which boots blazingly fast or you are using it bare metal, using an FPGA with embedded Flash (or using an external Flash IC) would be the only viable option. The SoC could then just update the Flash content if needed.

If the SoC intends to run Linux, the FPGA can be initialized within u-boot. u-boot being a bootloader means it does not require much to start up, and it also means that FPGA would be reliably booted up before kernel loads.

Interesting. I suppose that would entail having to meddle with u-boot? Or is it easily configurable? I don't know much about how u-boot works.

[technix]

Interesting. I suppose that would entail having to meddle with u-boot? Or is it easily configurable? I don't know much about how u-boot works.

Unless you are using kernel drivers for FPGA components, it is usually a better idea to defer FPGA configuration until you have a userland environment on SoC.

If you really want to do that, you need a u-boot driver that would write contents from a given memory address and length to the FPGA over some mechanism, for example SPI. Then existing u-boot mechanism allows you to load the bitstream file from a storage device, and invoke this driver to configure the FPGA. Research u-boot source code before coding your own though as that might have already been implemented.

[SiliconWizard]

OK thanks. Not something I would want to bother with I guess, unless I absolutely had to.

If I needed the FPGA to be ready early, I would just use a Flash chip I think, and just add means to reprogram the flash for updates...

[technix]

If I needed the FPGA to be ready early, I would just use a Flash chip I think, and just add means to reprogram the flash for updates...

You actually have a race hazard if FPGA and SoC boots individually: the SoC have little to no clue when would the FPGA actually finish loading the bitstream and booted, and should FPGA boot failed SoC would have little means to recover it. If you use SoC to bring up the FPGA, SoC code knows exactly when the FPGA is ready.

[mikeselectricstuff]

If I needed the FPGA to be ready early, I would just use a Flash chip I think, and just add means to reprogram the flash for updates...

You actually have a race hazard if FPGA and SoC boots individually: the SoC have little to no clue when would the FPGA actually finish loading the bitstream and booted, and should FPGA boot failed SoC would have little means to recover it. If you use SoC to bring up the FPGA, SoC code knows exactly when the FPGA is ready.

The SoC could have  an input to monitor the FPGA config state - you could have a system which normally boots the FPGA from flash to get some early functionality , and can be subsequently soft- reloaded  for the main functionality and/or the flash updated when the system has started

[legacy]

The SoC could have  an input to monitor the FPGA config state - you could have a system which normally boots the FPGA from flash to get some early functionality , and can be subsequently soft- reloaded  for the main functionality and/or the flash updated when the system has started

The DSP on the NOVA's board bootstrap an RTOS. Sure, the DSP and the FPGA do bootstrap singularly and in parallel, but there is a pin between the FPGA and the DSP; it's called "ready", and it means the FPGA has completed its initialization and it's running correctly. When the FPGA is power up the pin is down, and it only goes high only when the FPGA will complete the bootstrap and the finite state machine will have completed its early cycle. Here is where the HDL code plays its rules about initializing things. Besides, there are two latches programmed in the FPGA to tell about the hardware status. These is used by the "probe" functions in the kernel drivers.

Linux is more complex than an RTOS, but there is no problem at all 

[technix]

The SoC could have  an input to monitor the FPGA config state - you could have a system which normally boots the FPGA from flash to get some early functionality , and can be subsequently soft- reloaded  for the main functionality and/or the flash updated when the system has started

The DSP on the NOVA's board bootstrap an RTOS. Sure, the DSP and the FPGA do bootstrap singularly and in parallel, but there is a pin between the FPGA and the DSP; it's called "ready", and it means the FPGA has completed its initialization and it's running correctly. When the FPGA is power up the pin is down, and it only goes high only when the FPGA will complete the bootstrap and the finite state machine will have completed its early cycle. Here is where the HDL code plays its rules about initializing things. Besides, there are two latches programmed in the FPGA to tell about the hardware status. These is used by the "probe" functions in the kernel drivers.

Linux is more complex than an RTOS, but there is no problem at all 

READY pin provide no mechanism for recovering from a failed boot on the FPGA though. Any kind of slave port, be it JTAG or SPI, removes that point of failure as the SoC can recover the FPGA.

[NorthGuy]

Emulating 100 MHz QSPI flash is not an easy task. Real QSPI flash costs less than a dollar.

[SiliconWizard]

READY pin provide no mechanism for recovering from a failed boot on the FPGA though. Any kind of slave port, be it JTAG or SPI, removes that point of failure as the SoC can recover the FPGA.

So? Most circuits involvings FPGAs are designed like that. How often does an FPGA config from external flash fails? Not a huge point of concern. As long as you can detect the failure.

And as I suggested above, the simplest, fastest (configuration time wise) approach overall and still most flexible would be to use an external SPI/QSPI flash AND still implement JTAG.
The host can monitor the Config ready pin. If it doesn't show a ready state after a given time-out, the host can then reflash it through JTAG and/or proceed to some other automated testing.

So as I said before, if config time was critical, I personally would favor this approach for large bitstreams. For the smaller FPGAs, SPI slave would be probably adequate. Then again, as the OP suggested using MachXO3, if again it should be ready in a very short time, I would just buy the flash version, and implement reprogramming for updates through JTAG probably.

Again it's all in the requirements and use cases!
If, OTOH, config time is not critical, but you needed to be able to reconfigure the FPGA very often (maybe even "on the fly" depending on the system's state), obviously the SPI or JTAG only approach would be favored. Can also be a mix of both, and you can certainly do both with my suggestion of external (or internal if supported) flash + JTAG, and mikeselectricstuff's suggestion to monitor the config ready pin is definitely a good idea.

[technix]

Emulating 100 MHz QSPI flash is not an easy task. Real QSPI flash costs less than a dollar.

But I can DMA data from SoC memory to hardware SPI and have it piped out at ~120MHz if my SoC allows it. Ever heard of sendfile(2), can determine how to DMA files all within kernel mode?

[legacy]

READY pin provide no mechanism for recovering from a failed boot on the FPGA though. Any kind of slave port, be it JTAG or SPI, removes that point of failure as the SoC can recover the FPGA.

If the SPI-flash is programmed with the right bitstream, failed boots can only happen if there is an hardware problem, and there is anything the SOC can do to recover it in this case. The SOC can only issue a reset to the FPGA via GPIO, this will force the FPGA to reboot.