Multi microcontroller communication

[servoGuy]

Dear friends,

I would like to build some modular system to be able to use my previously designed hardware and stack boards up according to my needs. (e.g. anlog card, 24V IO card ...)
I am currently hanging at the protocol and hardware for the communication. I would definitely like to be flexible in the total number of boards I stack together so I would not like to use standard SPI which needs a slave select.

I thought about USART or SPI with ss always enabled. The protocol therefore would have to define some kind of identifier so the slave boards know when to respond.

Do you have any recommendations? What kind of physical layer is used in modern PLC-systems?

Looking forward to any recommendations or literature to get some inspiration.
 

[ataradov]

The problem with SPI for this application id that you will have a hard time convincing MCUs to not drive MISO line when SS is permanently enabled.

If super high speeds are not required, then I2C seems like a good option.

[ogden]

You shall check CAN bus. It could be what you are looking for.

[nctnico]

It depends a bit on how much intelligence is available on the boards. If the boards are inside the same (small enclosure) and there isn't more than 20cm of cable in total then I2C may be an option. Still I'd like to warn you that getting a distributed system up&running reliably is much harder than it seems. You'll need to account for the situation where a board doesn't respond at all and bring the system in a safe state nevertheless.

[rstofer]

Here is a general purpose multi-master protocol based on RS485 which is just a driver chip after the UART.
I suspect it might be overkill but it is about as general as a protocol can be.

http://bdmicro.com/code/robin/

I used this for a Halloween project one time and it worked well.

[servoGuy]

Thanks for the helpful replies. I thought about SPI because I need quite a high speed rate with 10Mbit/s.
For UART I would therefore need high oversampling (~speed * . CAN Bus might be an option if I use the newer FD CAN. However CAN also needs time segments and therefore kind of oversampling. Thats why I came up with using SPI.
I will do some small test setup trying to convince the MCU to only respond when the address bytes are correct although the slave select is tied high/low.

Every module will have a small MCU even for easy modules like IO, to implement the protocol.

Does anyone now what physical layer companies like Siemens, Beckhoff and others use for this purpose ?

[fchk]

Ethernet is a common choice now. Fast Ethernet (100MBit/s) is no problem at all with a small ARM (TI MSP432) or MIPS (PIC32MZ), and Gigabit Ethernet is often used with FPGAs dumping udp packets onto a PC. Realtime requirements require Ethernet exensions like EtherCat etc. which in turn require specialized MACs.

If data throughput and timing constraints are very tight you might not be able to use a bus at all but dedicated connections with differential signalling in a star topology.

Is the 10MBit/s data rate the upper limit for all devices on a bus or just a requirement for a single endpoint.

I2C is completely out - not suitable.

[David Hess]

fchk beat me to it.  The 10+ Mbit/s requirement on a shared bus makes it a reinvention of 10/100 shared media Ethernet but with a token being passed between the master and each slave so more like Token Ring without the virtual ring.

SPI and asynchronous serial can be that fast however almost always rely on programmed I/O which becomes a limitation at higher speeds.  Some USARTs have the ability to monitor for a specific address like a CAN controller does but it is not a common feature.

The CAN bus could be adapted to higher speed but I do not think the controllers will support it.

That leaves two options; reimplement shared 10/100 Ethernet without the transformers and use a standard Ethernet MAC which might be the easiest or use a slow shared bus like I2C, asynchronous serial, or CAN and a separate high speed synchronous bus if you can find a suitable USART.

Straight 10/100 Ethernet has by far the widest support so I would start there and implement it without the transformers on a shared bus with something like LVDS transceivers which are plenty fast.

[nctnico]

Why would you need 10Mbit things like analog and digital inputs. This typically is slow stuff.

[ogden]

Does anyone now what physical layer companies like Siemens, Beckhoff and others use for this purpose ?

Sure. PCI express

For lower end systems USB could be good idea to consider - due to vast number of various microcontrollers with built-in USB FS device peripheral.

[David Hess]

For lower end systems USB could be good idea to consider - due to vast number of various microcontrollers with built-in USB FS device peripheral.

The reason I did not recommend USB is that it cannot operate on a shared bus.  If point to point connections and a hub are acceptable, then USB might be a solution although I am not sure it would be better than Ethernet.

[ogden]

For lower end systems USB could be good idea to consider - due to vast number of various microcontrollers with built-in USB FS device peripheral.

The reason I did not recommend USB is that it cannot operate on a shared bus.  If point to point connections and a hub are acceptable, then USB might be a solution although I am not sure it would be better than Ethernet.

TP Ethernet is point-point and requires switch/hub as well. Please tell me which is better - nail or screw? Hammer or screwdriver? USB is easier to implement, much more popular in microcontrollers, needs less resources (flash/ram/power). On the other hand Ethernet is multi-master network which could be considered as huge advantage, also you may "spread" your modular device all over the place, not necessarily single chassis needed.

[David Hess]

TP Ethernet is point-point and requires switch/hub as well. Please tell me which is better - nail or screw? Hammer or screwdriver? USB is easier to implement, much more popular in microcontrollers, needs less resources (flash/ram/power). On the other hand Ethernet is multi-master network which could be considered as huge advantage, also you may "spread" your modular device all over the place, not necessarily single chassis needed.

10/100 Ethernet can operate over a shared media which is essentially what a hub, but not a switch, creates.  In the past this was commonly seen with 10BASE2.  A hub is required for 100 Mbit/s Ethernet because otherwise the stub length would be too long but this would not be a problem in an embedded application where the stub length can be controlled and very short.

Whether current 10/100 Ethernet MACs still support the functions required for CSMA/CD is the problem but since they still work with hubs, it seems likely.

[speakerguy]

At work we have legacy systems that use a 9-bit addressed UART scheme for multi-processor communications. Newer stuff is all CAN. Our data rates are not high though.

[SiliconWizard]

If your system allows one master only and several slaves, the simplest IMO would be to use RS485 or any LVDS phy interface on a 2-wire bus. The interface itself after the phy transceivers can be a simple UART. Just implement a protocol with at least the address of the slave in its header. Control the TX/RX state on the master and each slave according to the state it's in (listening or talking). That's straightforward. 10Mbit/s is no problem as long as your MCUs' UARTs allow it. A lot of RS485 transceivers can handle 20Mbit/s. LVDS transceivers can get you much higher than that as well.

If your system can be a multi-master topology, that gets trickier as you'll need to handle collisions and such. In that case, using an existing complete solution such as CAN would probably be easier.

[rstofer]

Thanks for the helpful replies. I thought about SPI because I need quite a high speed rate with 10Mbit/s.
For UART I would therefore need high oversampling (~speed * . CAN Bus might be an option if I use the newer FD CAN. However CAN also needs time segments and therefore kind of oversampling. Thats why I came up with using SPI.
I will do some small test setup trying to convince the MCU to only respond when the address bytes are correct although the slave select is tied high/low.

More often than not, it is the low going CS' signal that tells the peripheral to start listening and thus this signal frames the transaction.  Look very carefully at the timing diagrams for your MCU to see if tying CS' permanently low will even work.

High speed CAN runs between 40 Kbps and 1 Mbps per second depending on cable length.  This is easily in the range of RS485 and nowhere near 10 Mbps as required above.

https://knowledge.ni.com/KnowledgeArticleDetails?id=kA00Z0000019LzHSAU

In fact, if the UART is fast enough, RS485 can run up to 25 Mbps

http://www.ti.com/lit/ds/sgls367e/sgls367e.pdf

[rstofer]

It's not just the baud rate you need to consider on the MCU side.  I would assume that incoming chars are grabbed in an interrupt routine so preamble and postamble code are considerations.  In addition, there is the overhead of putting the chars into a circular queue.  Some MCUs have DMA and the DMA controller can manage a circular queue.  This would probably be limited to the more high end MCUs.

In gross terms, a serial UART stream of 10 Mbps interrupts at 1 MHz - 1 us per char.  I have no idea how fast the code will run without looking at the choice of MCUs.  A 16 MHz, 1 instruction per clock, probably isn't anywhere near fast enough.  Plus there is the pipeline flush on entry and return to consider.  Branch taken delay...

At the 10 Mbps data rate, there is a lot to consider.  When you need to worry about a pipeline flush, you're in serious trouble for speed.

Consider writing the interrupt handler in ASM or marking the interrupt C function 'naked' and doing your own preamble/postamble in inline assembly.  Every cycle is going to count!

If a transaction is very long or very frequent, will there be any compute cycles for the mainline?

[Richard Crowley]

It is not clear what level(s?) of communication you are talking about here.
The Open Systems Interconnect model has seven layers from physical through application.


You mentioned speed and hinted(?) at physical distance (adjacent?)
But did I miss average message length?  Or protocol, master/slave, contention, etc. requirements?

I was impressed that ROBIN seemed like a sensible protocol standard.

RJ45 connectors and UTP wiring are ubiquitous, dirt-cheap and quite high-speed.
And Ethernet-type magnetics (even built into RJ45 connectors) are great for isolation.
But that is just the Physical Layer and does not compel you to use TCP/IP or even CDMA/CD

[servoGuy]

To clarify my request. I plan a single master multiple slave network containing only MCU´s. I would like to easily physically add or remove slaves which is why I would like to not use a solid backplane with slots for add-on cards or similar. Instead the add-on cards shall each connect to the previous one beside an end card for termination etc. The total maximum number of add-on cards is 10.
Therefore I am looking for a communication interface which is commonly available on typical MCU´s like STM32, AVRs....
For the lowest, physical, layer M-LVDS, RS485 or similar should be sufficient.
10Mbit/s was a calculation with some margin for protocol overhead etc. One of the applications is a laser system with quite high update rates.

@David Hess: Ethernet would be an elegant solution and a field I am very familiar in. But most MCU´s have only one Ethernet MAC and I need that one for the communication with a linux computer. Furthermore Ethernet would lead to EtherCat which would mean each card would get twice as expensive.
Thats why I thought about SPI.

[max_torque]

Given that there is no requirement to provide a data bus with any significant "length" (ie you haven't got much distance between your slaves) then why not just use a basic parallel architecture. An 8b wide data bus, with 4 address lines would be trivial to implement, and pretty fast without having to run high slew rates / bit rates etc


Slaves could have a 4 way dip switch on them, setting their binary address, slaves could only transmit when the address lines are all lowered again by the master after the command message has been sent.

simple, flexible, and you could probably do the whole thing with raw microcontroller ports these days (no buffers)  and as your slaves are all local, it's trivial to provide say a 12b wide (+power) interconnection with say male/female pin headers allowing stacking of slaves etc

[servoGuy]

@max_torque: That might actually be an easy and very fast possibility. The only drawback is the number of connection pins between the slaves. Any recommendations for a connection header providing eg. 14pins  to easily connect modules?

[Psi]

I thought about USART or SPI with ss always enabled. The protocol therefore would have to define some kind of identifier so the slave boards know when to respond.

Some micros have "Multi-processor Communication Mode" or similar.
Multi-processor Communication Mode is what Atmel call it on ATmega chips.

It's basically some hardware that's part of the UART module which allows you to have an address ID with multiple micros on a bus and have them all ignore data (in hardware) unless its addressed to them.

Here's what the ATMega8 datasheet says about it.
I'm pretty sure its part of their full UART module so all ATMega chips should have it.

Multi-processor Communication Mode
Setting the Multi-processor Communication mode (MPCM) bit in UCSRA enables a filtering function of incoming
frames received by the USART Receiver. Frames that do not contain address information will be ignored and not
put into the receive buffer. This effectively reduces the number of incoming frames that has to be handled by the
CPU, in a system with multiple MCUs that communicate via the same serial bus. The Transmitter is unaffected by
the MPCM setting, but has to be used differently when it is a part of a system utilizing the Multi-processor Communication
mode.
If the Receiver is set up to receive frames that contain 5 to 8 data bits, then the first stop bit indicates if the frame
contains data or address information. If the Receiver is set up for frames with nine data bits, then the ninth bit
(RXB8) is used for identifying address and data frames. When the frame type bit (the first stop or the ninth bit) is
one, the frame contains an address. When the frame type bit is zero the frame is a data frame.
The Multi-processor Communication mode enables several Slave MCUs to receive data from a Master MCU. This
is done by first decoding an address frame to find out which MCU has been addressed. If a particular Slave MCU
has been addressed, it will receive the following data frames as normal, while the other Slave MCUs will ignore the
received frames until another address frame is received.


Using MPCM
For an MCU to act as a Master MCU, it can use a 9-bit character frame format (UCSZ = 7). The ninth bit (TXB8)
must be set when an address frame (TXB8 = 1) or cleared when a data frame (TXB = 0) is being transmitted. The
Slave MCUs must in this case be set to use a 9-bit character frame format.
The following procedure should be used to exchange data in Multi-processor Communication mode:
1. All Slave MCUs are in Multi-processor Communication mode (MPCM in UCSRA is set).
2. The Master MCU sends an address frame, and all slaves receive and read this frame. In the Slave MCUs,
the RXC Flag in UCSRA will be set as normal.
3. Each Slave MCU reads the UDR Register and determines if it has been selected. If so, it clears the
MPCM bit in UCSRA, otherwise it waits for the next address byte and keeps the MPCM setting.
4. The addressed MCU will receive all data frames until a new address frame is received. The other Slave
MCUs, which still have the MPCM bit set, will ignore the data frames.
5. When the last data frame is received by the addressed MCU, the addressed MCU sets the MPCM bit and
waits for a new address frame from Master. The process then repeats from 2.
Using any of the 5- to 8-bit character frame formats is possible, but impractical since the Receiver must change
between using n and n+1 character frame formats. This makes full-duplex operation difficult since the Transmitter
and Receiver uses the same character size setting. If 5- to 8-bit character frames are used, the Transmitter must
be set to use two stop bit (USBS = 1) since the first stop bit is used for indicating the frame type.
Do not use Read-Modify-Write instructions (SBI and CBI) to set or clear the MPCM bit. The MPCM bit shares the
same I/O location as the TXC Flag and this might accidentally be cleared when using SBI or CBI instructions.

[fchk]

@max_torque: That might actually be an easy and very fast possibility. The only drawback is the number of connection pins between the slaves. Any recommendations for a connection header providing eg. 14pins  to easily connect modules?

Look at PC/104:

http://suddendocs.samtec.com/catalog_english/esq_th.pdf

fchk

[NorthGuy]

Plain SPI should work fine as long as the master can drive the bus with 10 slaves, which may be Ok with relatively short distances at 10MHz. You can provide a single CS line for synchronization, then start every transmissions with an address of the slave you want to talk to. If the slave recognizes the address, it turns on its MISO line, or keep it tri-stated otherwise. This scheme also lets you have different slaves working at different speeds.

[zenman]

Have you considered using the ESP32 and transfer all the data wirelessly? This guy measured 13 Mbits/sec when using multiple threads: http://iot-bits.com/esp8266-tcp-server-speed-test/