SPI communication between two MCUs on same board

[jars121]

Afternoon,

I have two MCUs on a board; the first of which interfaces with a variety of analogue and digital components, and performs initial/low-level processing. This first MCU then passes the data over to the second MCU for final processing and a host of other functions.

I'm targeting the use of SPI for this process, as both MCUs support up to 60Mbits, and I can reuse various SPI drivers I've used in the past on both platforms. The second MCU has to be Master (a hard requirement of this particular MCU), which leaves the initial MCU as the Slave.

I'm trying to understand how this process would play out programmatically:

• I implement a 'status' or 'data ready' line between the first and second MCUs. The line is an output of the Slave MCU to indicate when new data is available to the Master.
• The Master commences an SPI transfer.
• The Slave keeps the status/data ready line asserted until the output FIFO is empty?
• The Master keeps sending out dummy data on MOSI until the status/data ready line is deasserted?

Does the above make sense? If not, would another approach be to toggle the status/data ready line, and have a fixed initial byte or byte sequence be a length indicator, so the Master knows how many bytes the Slave is going to send?

[fchk]

Building a Multi-Microcontroller system is almost always a bad idea. The communication makes the system so much more unstable an inefficient.

There are plenty of controllers with more then 200 MHz clock and more then 256k of RAM and hardware FPU. Just have a look at Cortex M4/M7 and PIC32MZ. There is even a new dual-core controller from NXP/Freescale with 1GHz M7 and 400Mhz Cortex M4, which communicate via shared memory. There is also STM32H7 with 480 MHz M7 and 240 MHz M4, and several others.

fchk

[jars121]

Thanks for your input fchk, it's much appreciated.

The Master MCU in this case is actually part of multi-core Cortex co-processor. I left this detail out on purpose as I specifically didn't want to focus on the 'should you have two MCUs on a board' and rather look at how this form of communication might be accomplished via SPI.

[Siwastaja]

Forget the crap about "multi-MCU is always bad and complex", sometimes it happens to be the right thing to do and saying it's always very complex and difficult tells more about the people saying that (and their skills). For example, I have no problem doing multi-MCU solutions, I sometimes do them when they help and not hinder. It depends. If in doubt, avoid, of course.

The big question is, what the data is?

Is it commands & acknowledges? Setpoints? Measurements?

Sometimes it is just as simple as setting up circularly running DMA to interchange piece of memory between the MCUs, very simple to set up, very robust, but there is no guarantee that the variables are updated at the same time i.e. there is no sync and atomicity is harder to set up, sometimes it matter sometimes it doesn't. For example, I have used this in motor controllers, so that the master shares speed and current limit setpoints, and the slaves share measured speed and current. No need for "sync".  Variable-level atomicity is achieved by configuring big enough SPI word (also DMA transfer) size, for example if your variables are at most 16 bits, use 16-bit SPI word and then the data in any variable is always valid. A simple increasing counter can detect a stuck connection, and CRC can be added for added robustness if you feel that way.

(People demanding a CRC being used for MCU-to-MCU SPI are, illogically, perfectly fine using pre-purchased external SPI ICs which implement no CRC. They are fine using all other board level digital signaling, for example to RAM or I2C sensors, without CRC, as well.)

On-board MCU-MCU is often simplified by the fact both CPUs are under your control, the firmware for both is written by the same person, and the CPUs are of the same architecture. Thusly, communication can reliably use packed structs, and both firmware projects can share the same header file.

[jpanhalt]

We might be able to give more specific advice, if we knew the MCU's. 

I am mostly familiar with PIC's.  In that example for SPI, the PIC that sets the other device's CS input low is "master."  The input pins are defined in the datasheets.  The pin you use to do that with the Master can be any I/O pin.  So,#1 MCU can control #2 by using #2's CS.  #2 can control a peripheral (e.g., sensor, LCD) using any output to that peripheral's CS input.  Some peripherals may require additional pins to control whether the data are commands, writes, or reads.

[ledtester]

I'm targeting the use of SPI for this process, as both MCUs support up to 60Mbits, ...

I'm trying to understand how this process would play out programmatically:

• I implement a 'status' or 'data ready' line between the first and second MCUs. The line is an output of the Slave MCU to indicate when new data is available to the Master.
• The Master commences an SPI transfer.
• The Slave keeps the status/data ready line asserted until the output FIFO is empty?
• The Master keeps sending out dummy data on MOSI until the status/data ready line is deasserted?

Does the above make sense? If not, would another approach be to toggle the status/data ready line, and have a fixed initial byte or byte sequence be a length indicator, so the Master knows how many bytes the Slave is going to send?

I would consider using serial communication (i.e. via a UART) between the two mcus - then each side can send a packet of data whenever they want/need to. That would greatly simplify the protocol.

What processors for the master and slave are involved here?

Update: SPI is fast because it is also very simple. It was initially designed to be used with shift registers. MOSI data from the master simply entered into a shift register on the slave which after so many bits was latched into the slave's control register. MISO data from the slave also came from a shift register that the master just clocked out. It wasn't designed for "send data and wait around for a response" type transactions.

So I would use SPI only for uni-directional transfers. A multi-master SPI bus is a possibility, or use two dedicated communication channels (one for TX and another for RX) between the processors like the UART idea.

[SiliconWizard]

Software isolation using separate processors may also be a good thing. This is often used in safety-critical applications. It's very hard to ensure proper isolation within the same processor, especially if everything runs on a single core, but even if not, cores usually have to share memory, so full isolation is anywhere from hard to almost impossible.

As to communication, it adds risks certainly, but apart from the additional hardware issues, it's not that much more difficult than robust communication between different tasks on the same processor.

Of course, if the system is simple enough, that may not make sense at all.

But as always, only proper system analysis can help you determine whether it is worth it or not, rather than using hard rules and prejudice.

[NorthGuy]

Forget the crap about "multi-MCU is always bad and complex", sometimes it happens to be the right thing to do and saying it's always very complex and difficult tells more about the people saying that (and their skills).

Sure there may be cases where using two MCUs instead of one is beneficial, but such cases are rare and far between.

When you use two MCUs you need to find a way to program both, you need to maintain sources for both, you need to organize a communication protocol. With a single MCU you don't need any of that. So, there must be a compelling reason to justify extra efforts for using two MCUs. I'm sure, every time Siwastaja uses multi-MCU design, he has his reasons.

IMHO, a wise thing to do before designing a protocol is to ask: "What is the reason two MCUs are used instead of one"?

[jars121]

Thank you everyone for your input, it is very much appreciated and has made for a good read. Before I respond to some individual posts, I want to reiterate that the multi-MCU requirement is not the subject of this question. As has been articulated in several responses now, sometimes there are valid reasons for using multiple MCUs or an MCU with an external processor; this happens to be one of those scenarios.

Forget the crap about "multi-MCU is always bad and complex", sometimes it happens to be the right thing to do and saying it's always very complex and difficult tells more about the people saying that (and their skills). For example, I have no problem doing multi-MCU solutions, I sometimes do them when they help and not hinder. It depends. If in doubt, avoid, of course.

The big question is, what the data is?

Is it commands & acknowledges? Setpoints? Measurements?

Sometimes it is just as simple as setting up circularly running DMA to interchange piece of memory between the MCUs, very simple to set up, very robust, but there is no guarantee that the variables are updated at the same time i.e. there is no sync and atomicity is harder to set up, sometimes it matter sometimes it doesn't. For example, I have used this in motor controllers, so that the master shares speed and current limit setpoints, and the slaves share measured speed and current. No need for "sync".  Variable-level atomicity is achieved by configuring big enough SPI word (also DMA transfer) size, for example if your variables are at most 16 bits, use 16-bit SPI word and then the data in any variable is always valid. A simple increasing counter can detect a stuck connection, and CRC can be added for added robustness if you feel that way.

(People demanding a CRC being used for MCU-to-MCU SPI are, illogically, perfectly fine using pre-purchased external SPI ICs which implement no CRC. They are fine using all other board level digital signaling, for example to RAM or I2C sensors, without CRC, as well.)

On-board MCU-MCU is often simplified by the fact both CPUs are under your control, the firmware for both is written by the same person, and the CPUs are of the same architecture. Thusly, communication can reliably use packed structs, and both firmware projects can share the same header file.

Thanks Siwastaja. In answer to your question, the Slave MCU is passing measurement data to the Master. On booting, the Master sends configuration data to the Slave, which it may update during runtime. However, for the most part, the Slave is sending measurement data. I had envisioned (and have used elsewhere) a similar circular DMA setup; the measurement data itself carries some metadata which means the sync and atomicity components perhaps aren't that important.

I would consider using serial communication (i.e. via a UART) between the two mcus - then each side can send a packet of data whenever they want/need to. That would greatly simplify the protocol.

What processors for the master and slave are involved here?

Update: SPI is fast because it is also very simple. It was initially designed to be used with shift registers. MOSI data from the master simply entered into a shift register on the slave which after so many bits was latched into the slave's control register. MISO data from the slave also came from a shift register that the master just clocked out. It wasn't designed for "send data and wait around for a response" type transactions.

So I would use SPI only for uni-directional transfers. A multi-master SPI bus is a possibility, or use two dedicated communication channels (one for TX and another for RX) between the processors like the UART idea.

Thanks ledtester. UART would make perfect sense, but I require much higher throughput than is supported by UART. Somewhere around the 20Mbit mark would be great, but with some headroom to expand in the future. I understand your point about SPIs simplicity. That's one of the reasons I'm drawn to it. As mentioned above, the data being sent carries some metadata, so the 'complexity' of the transmission is removed from the underlying hardware/interface.

You've raised a valid point, which I hadn't considered. Perhaps I use the high-speed SPI as a unidirectional mechanism from the Slave to the Master, and have a secondary mechanism (such as UART) for passing of configuration from the Master to the Slave, and ACKs/NACKs of the sent configuration from the Slave to the Master. It may simplify the processing if the SPI interface is used only for the high speed measurement information.

Sure there may be cases where using two MCUs instead of one is beneficial, but such cases are rare and far between.

When you use two MCUs you need to find a way to program both, you need to maintain sources for both, you need to organize a communication protocol. With a single MCU you don't need any of that. So, there must be a compelling reason to justify extra efforts for using two MCUs. I'm sure, every time Siwastaja uses multi-MCU design, he has his reasons.

IMHO, a wise thing to do before designing a protocol is to ask: "What is the reason two MCUs are used instead of one"?

Thanks NorthGuy. As I raised earlier, I spent months evaluating a host of hardware configurations, and the MCU + co-processor option was the most suited to this particular application.

[lucazader]

We use multiple MCUs on single boards, and in single products at my work.

The way we configure the comms is low speed data ( less than `8mbit) and configuration channels are done with a UART.
Highspeed data is done over SPI in a much more directional manner that can be configured by the associated uart channel.

[nctnico]

Thanks for your input fchk, it's much appreciated.

The Master MCU in this case is actually part of multi-core Cortex co-processor. I left this detail out on purpose as I specifically didn't want to focus on the 'should you have two MCUs on a board' and rather look at how this form of communication might be accomplished via SPI.

You have to be aware that SPI is highly susceptible to interference especially on the clock line. Running SPI at high speeds increases this problem because you can't add too much filtering (unlike I2C and UART which have been designed for communication over longer distances, SPI has no integrated filtering of noise). All in all you will have to incorporate CRC checks in the communication protocol when using SPI and preferably in any type of physical communication layer which doesn't have some form of CRC checks integrated. If you are using reasonably high-end microcontrollers using the ethernet MAC can be a better alternative because it runs the lines at a lower speed which allows more filtering on long traces. You can clock a MAC at any speed you want; just as long as it is enough to fullfill the bandwidth requirements. You can tie the MACs together without a PHY and magnetics in between. The advantage is that you have a well defined packet based messaging layer between the two controllers with CRC checks, full DMA controlled transfers (usually) and lots of bandwidth.

But like others I'm also in the camp of 'use 1 microcontroller if you can'. Remember most of the brave heroes are dead. You will need to cater to error situations in which messages are lost, there is no fixed time synchronisation, protocol versions are different and one of the microcontrollers can become non-responsive. The less tight the integration is, the lesser you will be affected by these problems but requiring a high bandwidth makes me wary that you are working on a distributed system where several controllers need to work together very tightly. Don't discard this warning lightly! Your question about whether SPI is suitable makes me wonder whether you have designed a multi-microcontroller system before. During my career I have re-designed several systems where people severely underestimated the difficulty of getting multiple microcontrollers to work together.

As someone else has pointed out: NXP has these very powerful cross-over microcontrollers which run at speeds up to 1GHz.

[NorthGuy]

Thanks NorthGuy. As I raised earlier, I spent months evaluating a host of hardware configurations, and the MCU + co-processor option was the most suited to this particular application.

I don't know how you made this determination without considering protocols.

Anyway. I would make the MCU which sends ADC data an SPI master, and the MCU which receives it an SPI slave - with 3 wires - CS, CLK, MOSI. It'll work better than the other way.

As other suggested, I would also add two separate UART lines (TX and RX) for control communications and such.

[jars121]

We use multiple MCUs on single boards, and in single products at my work.

The way we configure the comms is low speed data ( less than `8mbit) and configuration channels are done with a UART.
Highspeed data is done over SPI in a much more directional manner that can be configured by the associated uart channel.

Thanks lucazader. This is very promising to hear. I'll give this some further thought, as I can definitely see this working.

You have to be aware that SPI is highly susceptible to interference especially on the clock line. Running SPI at high speeds increases this problem because you can't add too much filtering (unlike I2C and UART which have been designed for communication over longer distances, SPI has no integrated filtering of noise). All in all you will have to incorporate CRC checks in the communication protocol when using SPI and preferably in any type of physical communication layer which doesn't have some form of CRC checks integrated. If you are using reasonably high-end microcontrollers using the ethernet MAC can be a better alternative because it runs the lines at a lower speed which allows more filtering on long traces. You can clock a MAC at any speed you want; just as long as it is enough to fullfill the bandwidth requirements. You can tie the MACs together without a PHY and magnetics in between. The advantage is that you have a well defined packet based messaging layer between the two controllers with CRC checks, full DMA controlled transfers (usually) and lots of bandwidth.

But like others I'm also in the camp of 'use 1 microcontroller if you can'. Remember most of the brave heroes are dead. You will need to cater to error situations in which messages are lost, there is no fixed time synchronisation, protocol versions are different and one of the microcontrollers can become non-responsive. The less tight the integration is, the lesser you will be affected by these problems but requiring a high bandwidth makes me wary that you are working on a distributed system where several controllers need to work together very tightly. Don't discard this warning lightly! Your question about whether SPI is suitable makes me wonder whether you have designed a multi-microcontroller system before. During my career I have re-designed several systems where people severely underestimated the difficulty of getting multiple microcontrollers to work together.

As someone else has pointed out: NXP has these very powerful cross-over microcontrollers which run at speeds up to 1GHz.

Thanks nctnico. It's funny that you mention the ethernet approach. I have a version of this design which uses exactly that. I use the ethernet interface of both MCUs tied directly together, and built a full duplex TCP-based communication mechanism. It was indeed reliable, but the maximum throughput I was able to achieve was ~10Mbit. The overhead of the ethernet driver on the Slave MCU, coupled with the overhead of the TCP protocol meant I wasn't getting the throughput I needed.

I don't know how you made this determination without considering protocols.

Anyway. I would make the MCU which sends ADC data an SPI master, and the MCU which receives it an SPI slave - with 3 wires - CS, CLK, MOSI. It'll work better than the other way.

As other suggested, I would also add two separate UART lines (TX and RX) for control communications and such.

Per the above comment, the determination was actually made based on using the ethernet approach. I'm unsure whether I'm able to make the receiving MCU an SPI Slave; I'll have to look into that further. If I am able, your approach makes perfect sense.

[nctnico]

You have to be aware that SPI is highly susceptible to interference especially on the clock line. Running SPI at high speeds increases this problem because you can't add too much filtering (unlike I2C and UART which have been designed for communication over longer distances, SPI has no integrated filtering of noise). All in all you will have to incorporate CRC checks in the communication protocol when using SPI and preferably in any type of physical communication layer which doesn't have some form of CRC checks integrated. If you are using reasonably high-end microcontrollers using the ethernet MAC can be a better alternative because it runs the lines at a lower speed which allows more filtering on long traces. You can clock a MAC at any speed you want; just as long as it is enough to fullfill the bandwidth requirements. You can tie the MACs together without a PHY and magnetics in between. The advantage is that you have a well defined packet based messaging layer between the two controllers with CRC checks, full DMA controlled transfers (usually) and lots of bandwidth.

But like others I'm also in the camp of 'use 1 microcontroller if you can'. Remember most of the brave heroes are dead. You will need to cater to error situations in which messages are lost, there is no fixed time synchronisation, protocol versions are different and one of the microcontrollers can become non-responsive. The less tight the integration is, the lesser you will be affected by these problems but requiring a high bandwidth makes me wary that you are working on a distributed system where several controllers need to work together very tightly. Don't discard this warning lightly! Your question about whether SPI is suitable makes me wonder whether you have designed a multi-microcontroller system before. During my career I have re-designed several systems where people severely underestimated the difficulty of getting multiple microcontrollers to work together.

As someone else has pointed out: NXP has these very powerful cross-over microcontrollers which run at speeds up to 1GHz.

Thanks nctnico. It's funny that you mention the ethernet approach. I have a version of this design which uses exactly that. I use the ethernet interface of both MCUs tied directly together, and built a full duplex TCP-based communication mechanism. It was indeed reliable, but the maximum throughput I was able to achieve was ~10Mbit. The overhead of the ethernet driver on the Slave MCU, coupled with the overhead of the TCP protocol meant I wasn't getting the throughput I needed.

Using TCP protocol seems like total overkill to me (IOW: TCP is the wrong choice for an application like this). With the MACs tied together you can use raw ethernet frames and basically feed a buffer into it directly which comes out in the same order at the other side.

[jars121]

Using TCP protocol seems like total overkill to me (IOW: TCP is the wrong choice for an application like this). With the MACs tied together you can use raw ethernet frames and basically feed a buffer into it directly which comes out in the same order at the other side.

That's a fair point, TCP is overkill for this application.

Given that I've already got hardware available with the MACs tied together I can certainly test the raw ethernet frame approach. Do you have any thought as to potential bandwidth improvement by adopting this approach over using TCP?

[nctnico]

Using TCP protocol seems like total overkill to me (IOW: TCP is the wrong choice for an application like this). With the MACs tied together you can use raw ethernet frames and basically feed a buffer into it directly which comes out in the same order at the other side.

That's a fair point, TCP is overkill for this application.

Given that I've already got hardware available with the MACs tied together I can certainly test the raw ethernet frame approach. Do you have any thought as to potential bandwidth improvement by adopting this approach over using TCP?

That depends entirely on which TCP/IP stack you are using, how it is interfaced to the rest of the software and what the rest of the software is doing. If you can buffer data into large packets and have DMA take care of the actual transmission then the overhead for the communication is very low.

[jars121]

I'm afraid I misspoke earlier. The current TCP-based design does use a PHY on both ends. The Slave MCU has a standard, external 10/100 PHY, while the Cortex-based side has an integrated 10/100/1000 PHY. Can a raw ethernet frame approach be used given that there are PHYs involved?

[Nominal Animal]

If you treat the SPI bus transfers as (larger) fixed-size packets, this is workable.

For example, let's say each packet is 296 bits (8 bit flags, 256 bit data, 32 bit CRC), the SPI clock is about 50 MHz (48 MHz effective), and both ends use DMA.
The DMA completion interrupt will then fire up to 160,000 times per second, and the bandwidth exceeds 40 Mbit/s.  I would have the master send padding packets whenever it has no data to send, so that the SPI clock is basically continuous; that way you don't need a separate "slave has data" line, and you minimise the latencies.

The flags in each packet sent – including no-data/all-zero packets – contain a RESEND (ACK/NAK) flag for the packet received before the last one, i.e. there is always at least one packet "in flight".  This means that SPI data can flow all the time, and does not need to stop while the DMA completion interrupt verifies whether the packet is correct or not; the next packet to be sent is already known/constructed when the DMA completion interrupt fires.  It also means that the buffer cannot be reused immediately after it is sent; each end needs at least three packets' worth of buffer per direction.

The data in each packet itself can be a stream, or consist of logical packets.  The above is basically just the hardware layer, somewhat similar to raw ethernet frames.

[Nominal Animal]

Can a raw ethernet frame approach be used given that there are PHYs involved?

Sure; why not?

Instead of an IP stack on top, you just need to create your own format for the payload, with an ACK/RESEND mechanism to resend garbled frames.

[NorthGuy]

I'm afraid I misspoke earlier. The current TCP-based design does use a PHY on both ends. The Slave MCU has a standard, external 10/100 PHY, while the Cortex-based side has an integrated 10/100/1000 PHY. Can a raw ethernet frame approach be used given that there are PHYs involved?

Ethernet let devices exchange frames. The IP protocol is built on top of Ethernet frames and includes frame format and addressing (IP addresses and ports). The TCP protocol is built on top of IP and lets you create peer-to-peer connections and manage them. I don't think why you would need either TCP or IP. Just use raw Eithernet frames. There's some overhead, but you certainly can achieve 10 MB/sec full duplex over 100 Mbps link.

[jars121]

Brilliant, thank you both. This has given me plenty to research, think about and test. As you can tell, I have very little experience in lower level networking, so I apologise for the very basic clarifications.

Ethernet let devices exchange frames. The IP protocol is built on top of Ethernet frames and includes frame format and addressing (IP addresses and ports). The TCP protocol is built on top of IP and lets you create peer-to-peer connections and manage them. I don't think why you would need either TCP or IP. Just use raw Eithernet frames. There's some overhead, but you certainly can achieve 10 MB/sec full duplex over 100 Mbps link.

Just to be absolutely clear, that's 10MBytes/sec correct?

[NorthGuy]

Just to be absolutely clear, that's 10MBytes/sec correct?

Sure. TCP/IP adds more overhead, but it shouldn't be too far off neither. If you got substantially less then either your link was not 100 Mbps, or you added a lot of overhead somewhere else.

[jars121]

Sure. TCP/IP adds more overhead, but it shouldn't be too far off neither. If you got substantially less then either your link was not 100 Mbps, or you added a lot of overhead somewhere else.

I was using the FreeRTOS +TCP stack, and was seeing around 10Mbit/s RX and 18Mbit/s TX using iPerf; these figures were corroborated by other FreeRTOS users on this platform. There is definitely room to optimise further, but as has been stated earlier, the use of the full TCP/IP stack is complete overkill for a (largely) unidirectional, point to point channel.

[Siwastaja]

UART has "packet" size of at most 8-10 bits (usually). It's delimited by start and stop bits decoded by the peripheral.

SPI has arbitrary packet size, many kilobytes are possible. It's delimited by the hardware nSS signal, hopefully decoded by the peripheral but beware, some SPI peripheral implementations on some MCUs suck.

"Packet" refers here to the hardware-provided delimitation. This is highly important; if you don't have it on hardware, you need to write your own. Conversely, if you have it provided for you, you don't need to reinvent the wheel, and your code base will be simpler and more robust.

So it's always simpler if your higher-level communication can use the hardware-provided low-level delimitation directly. With SPI, this is possible. With UART, you need to write your own protocol for delimitation in most cases (only rarely is one byte per communication enough); minimum workable design consists of start byte and packet length. With SPI, this isn't needed. So SPI may be simpler than UART! It depends.

TCP is like UART, it's called a stream of bytes (octets), "packet" is 8 bits (fixed size). Underlying hardware packets are abstracted away. As with UART, 8 bit commands / data is rarely usable, so you need to build higher level message delimitation on top, using something premade or writing your own. This is extra work, but with TCP, you can work reliably with the very simple messaging schemes because TCP is guaranteed to never lose, duplicate or corrupt a byte. But getting a TCP stack running on MCU just to share a bit of data between two controllers sounds like killing a fly with a cannon.

SPI isn't any more susceptible to interference than any other CMOS push-pull communication link. If SPI fails, external RAM or flash fails as well. Yet nctnico is fine using external RAM or FLASH, but not fine using SPI. This is highly illogical (and discussed to death already). Yet real computer systems do, for example, run the program out of external RAM where a single induced clock system kills the whole system. The usual way is to look at the root cause of what is causing the signal integrity issues, and correct the SI.

Special hardened systems combine good signal integrity with extra measures for error correction and recovery, but it seem nctnico is substituting signal integrity with error detection and recovery and only doing that with SPI leaving all other CMOS links susceptible to bad SI. Go figure.

UART does have extra robustness built in because the typical MCU UART peripheral samples the signal 8 or 16 times per bit. SPI and other clocked CMOS buses just clock the data bits once on the clock edge, so signal integrity is indeed important. This isn't an excuse for accepting poor signal integrity, though, and it doesn't take much extra SI problems to kill UART, as well.

Note that communication schemes that just "spam" the data are easiest to deal with; for example automotive systems (that utilize CAN as the communication layer, incidently designed for such scheme) work by repeating the measurements, settings etc. on the bus frequently enough; intended receivers implement timeouts in case they don't get the important value they need. Same idea can be built on SPI. Even if you have a rare incidence of severe noise corrupting the transfer (for example, coming from a severely non-compliant device) causing a missing or an extra clock cycle, the flow control doesn't go haywire because each packet is delimited by the hardware nSS; at most, that particular packet fails. Proper SPI slave resets its rx state when nSS goes to inactive state, starting receiving the next one correctly. Adding CRC is not a bad idea if you are working with a critical system in iffy conditions. Obviously, do that with UART or any other physical layer, as well. CAN offers this on the protocol/peripheral level for you.

[SiliconWizard]

SPI is definitely no problem if both MCUs are on the same board (and traces are short between them.)

As Siwastaja noted, if it was problematic, then you would never use any SPI peripheral, nor any external memory.

Then it obviously largely depends on proper signal routing and the clock speed you want to achieve.

If you are really concerned about signal integrity and/or the two MCUs are going to be "far" apart, you can always use LVDS. That's still less of a hassle than implementing ethernet and a full TCP stack.
Now if you're not going baremetal but are already using some kind of OS, and said OS already supports Ethernet/TCP, then it would likely make more sense to use Ethernet. Solutions always depend on requirements and context.

As to data integrity, if you're going the simple path of SPI (or UART), you can implement a relatively simple protocol with acknowledge packets and CRCs. That's not rocket science.

[nctnico]

SPI isn't any more susceptible to interference than any other CMOS push-pull communication link. If SPI fails, external RAM or flash fails as well. Yet nctnico is fine using external RAM or FLASH, but not fine using SPI. This is highly illogical (and discussed to death already).

External RAM and flash are not synchronous like SPI and usually RAM & flash are very close to the processor so interference is unlikely. Remember a false pulse on the SPI clock line means a state change! If you are going to route SPI over a board for 10 cm or more you can run into trouble. Especially with high power components nearby (even on 4 layers boards with a solid ground plane). I'm not pulling this out of my ass; this is hands on experience talking. You can keep yabbering on about signal integrity but the fact is that SPI is more prone to problems over long traces compared to I2C.

If you compare to I2C you'll see the electrical interface of I2C has been specifically designed to filter out spikes and offer a large noise margin. As a result you can run an I2C bus over tens of cm on a single layer board inside a CRT TV without problems. I2C is not a simple push-pull communication link, it has been cleverly designed to have a low susceptibility against noise / interference. The same goes for UARTs. These also have been designed to deal with transmission in noisy environments.

[NorthGuy]

I was using the FreeRTOS +TCP stack, and was seeing around 10Mbit/s RX and 18Mbit/s TX using iPerf; these figures were corroborated by other FreeRTOS users on this platform. There is definitely room to optimise further, but as has been stated earlier, the use of the full TCP/IP stack is complete overkill for a (largely) unidirectional, point to point channel.

Look for a bottleneck. Data transmission is always a chain of events. Find the event which is the slowest. Such event will hold the whole chain and will limit the speed of the whole transmission.

There could be a software bottleneck (some part of the software is too slow), or hardware bottleneck (e.g. a controller is too slow).

[jars121]

Thanks all for your input.

Signal integrity has been raised a number of times now, which is a valid concern. I have other SPI interfaces in use elsewhere on the board, which have been tested at 50MHz with no issues. I don't expect there to be any issues with signal integrity if I were to use SPI for this particular interface, as I'll pay the same very close attention to routing and ensuring all best-practise SI considerations are followed.

Having said that, I've done some further reading overnight, and am investigating the raw ethernet frame approach a little more closely. Both PHYs support auto-negotiation by default, so a 100BASE-TX full-duplex link should automatically be established. This should mean I don't need a PHY driver, but it has been suggested it might still be a good idea to have a light-weight PHY driver to monitor the link status, to know when the link has been established, etc.

I'm still not entirely clear on the driver for the raw ethernet frame. I suppose I can start with the TCP/IP stack I already have, that currently works, and work backwards from the point of data being sent out on the wire.