Suggestions For Communication Between Multiple FPGAs

[petersanch]

Hi All, what are some suggestions for communication protocols between multiple FPGAs so that any FPGA can address and communicate to any other FPGA?

Considering multi drop bus or daisy chain bus using IO pins directly. Are there common protocols used in FPGAs for this reason without writing something as complex as a TCP stack?


Cheers

[tggzzz]

That is far too vague to give a useful answer.

You have to define the characteristics of the (hardware) processes generating and processing the information, plus the required system behaviour and the fault conditions it has to deal with.

[Bassman59]

Hi All, what are some suggestions for communication protocols between multiple FPGAs so that any FPGA can address and communicate to any other FPGA?

Considering multi drop bus or daisy chain bus using IO pins directly. Are there common protocols used in FPGAs for this reason without writing something as complex as a TCP stack?

What are your data rate requirements? That'll tend to drive your choices.

So too will the number of pins you can use for this.

The great thing about FPGA-to-FPGA transfer like you describe is that you can use a protocol that makes sense for the application, without having to shoehorn in something that was designed for something else.

[petersanch]

Number of pins is flexible. 8 bit or 16 bit bus is ok. Data requirements are preferred between 50- 100 Mbytes/sec. Want to be able to address up to 10 FPGAs in the "network".
Packet collision is an obvious problem in a multi drop bus. Packet routing is more cumbersome in a daisy chain bus. Do you recommend one over the other?

For now I want any FPGA to be able to send data to any other FPGA. Requesting data is for the future.

Cheers

[Someone]

With hard serdes on most FPGA pins now the limiting factor is your physical interface.

Data rate vs routing scheme vs encoding vs .. other considerations. It not something you just come up with in a short question.

[NiHaoMike]

Number of pins is flexible. 8 bit or 16 bit bus is ok. Data requirements are preferred between 50- 100 Mbytes/sec. Want to be able to address up to 10 FPGAs in the "network".
Packet collision is an obvious problem in a multi drop bus. Packet routing is more cumbersome in a daisy chain bus. Do you recommend one over the other?

You want LVDS links point to point between the FPGAs. If you can designate one FPGA as the master, you could do a star topology. Other good choices are mesh (most likely partial since a complete mesh with 10 nodes requires a lot of links!) and ring.

[TomS_]

Is Ethernet out of the question? You can implement your own layer 2 protocol and just use ethernet for the network portion, taking care of switching frames where they need to go etc since that is already a very well established process.

But then maybe you have to shoehorn one part or another to make that fit your needs, which as others have suggested might not be ideal.

[asmi]

For Xilinx 7 series and above, there is a free "AXI Chip2Chip" IP, which can work over just about any PHY - single-ended, differential (LVDS), or multi gigabit transceivers (MGT). It is bidirectional and has configurable width, allowing to reach bandwidth of gigabits per second while having relatively low latency. For example, 64bit AXI bus with 4-1 serialization over 28 IO pins 150 MHz DDR PHY has a peak throughput of 1780 Mb/s with 95 cycles of latency for writes, and 1920 Mb/s and 59 cycles of latency for reads.  Search for "PG067" on Xilinx website for details.

[laugensalm]

Number five would say: Need more input.
Like: What's the distance between FPGA, what kind of data, packet bursts, or continous stream, blanks, no blanks (then you need to care about clock synchronization, possibly).
I tend to use Gigabit Ethernet, just for being able to debug or communicate with other hardware or being able to just buy any off-the-shelf FPGA board and not having to design anything niche-y. A tiny soft core sets up a DMA packet queue, the rest is done by the hardware (like streaming from HDMI to RTP/UDP). But you could also run some simple LVDS over serial ATA cables or HDMI when one way.
Best is probably to just pick up any working reference that can be simulated thoroughly. Bugs will cost time, even if it's a simple protocol.

[tggzzz]

Indeed.

The OP has not responded to reply #1, probably because he doesn't understand the question.

Perhaps putting it another way, in network protocol terms...

Does the OP want or need a "protocol" at OSI model protocol level 1,2,3,4,5,6,7? https://en.wikipedia.org/wiki/OSI_model

[BrianHG]

Well, you can try super dumb if all the FPGAs are on 1 PCB and you do not need to worry about errors & you have additional IOs for flow control.  I would use the FPGA's built in serdes in 16 bit mode.  I would use a header word to define the target destination(s) + packet size.  And additional normal IO per FPGA in open drain mode for outbound traffic control.  If the serdes is fast enough you may change the open-drain IO for a sequential counter just giving each FPGA 1 time slot to become the outbound master with a dumb defined lot of time.  If you only have half or 1/4 the total speed you need when doing this, use 2 or 4 serdes in parallel to make up the difference.

I would carefully tie each serdes to 1 dual port ram block, obviously making that block 1 exact block size for maximum cache with minimum core blocks, divided into all the source and destination banks, with a logic based R/W ready/ack & buffer stale flag for each target block making a minimal system to automatically share a com chunk/buffer cross shared between each FPGA in the system.

Like I said, this is the dumb no protocol method, just each FPGA would have a window of ram in each other being refreshed as needed.  This takes no intelligence, not protocol or handshaking to follow.

[Guy Shemesh]

I agree with everyone above me who said it depends on the data requirements. I assuming for a moment that the data may be transferred as bursts and is not continuous.

So, to keep things simple I would use a circular daisy chain with in/out streams on each FPGA. Each stream comes with 4 data, a clock and a data valid signal. Total 6 wires per stream and 12pin per FPGA. The data is clocked as DDR, this is OK for 50+MB/S on the same board (and depending on the FPGA you could probably have 100MB/S as well).

On the FPGA you just implement a BRAM as fifo in-out and also tap the data to another fifo - in case the data is addressing your FPGA. The protocol is very simple, first byte is the FPGA address or opcode description which would tell the FPGA if the data should be tapped.

In case you reach a full circle you simply discard the data.

In case you need to send new data on the line you first have to wait until the in-out fifo is empty to certain threshold - in order to be able to receive a new packet in case one arrives while you transmit your data. This creates a relationship between your fifo size and maximum packet size.

I would prefer this because of the simplified PCB design and further it doesn't require expensive FPGAs. And you could also implement a data logger who just listens on the line and routes the data to another medium such as ethernet or usb.

If you need help with this please feel free to reach out, we provide electronic engineering services including FPGA.

(I hope it ok to publish here, if not I'll remove this)

Best of luck
Guy