RS485 communication handling

[Simon]

I am working on a system where I have a machine ECU and a HMI that will talk over RS485.

This means of course that there must be some sort of protocol and message handling, currently I detect a received byte and start putting the bytes into a buffer array in the interrupt handler each time the interrupt fires for a byte received. But how do I guard against any spurious data coming in like some transition on the bus when the other device starts up? is such an event likely?

I can think of 2 methods 1) have a start byte, something like 0x55 (01010101) should be fairly demanding in satisfying the timing requirements to be picked up as a byte that should be taken seriously. 2) I know that a message should be at least say 4 bytes so if I don't get at least 4 bytes in a timely fashion I ignore it all and reset by buffer index to 0.

[pcprogrammer]

You can use a CRC on the data and reject messages that fail the test. But you will have to make up some protocol anyway because you are controlling something from a human machine interface. So a preamble like 0x55 followed by a message length and at the end the CRC can work.

Another option is to use a longer preamble like 0x55 0xAA and only receive more data when such a sequence has been received. This reduces the problem of 0x55 being part of a valid message. The change of 0x55 0xAA being part of a valid message is less.

[woofy]

Data corruption can and does happen in an industrial environment. It might only be a single bit in one byte but its enough to ruin your day.

Use an interframe gap as the trigger for a new frame, or bit 8 (9th data bit) of the data if that is available to you. Try not to trigger off a specific byte, and don't rely on having "enough bytes".
For the protocol divide the frame into a header/payload and CRC.
The header will contain at least the target address (if there is any possibility of more than one device on the bus) and the payload count.
That is followed by the application payload, and finally the CRC of the entire frame - up to the CRC of course.

Its the CRC that will protect your data, if there is a mismatch then throw the frame away.

[Simon]

Yes the plan is a message format with an address and error check at the end. My concern is that if a reception is triggered by anything I could never be in sync. So say some junk happens and the receiver takes this to be the first byte then it is stored in the first byte of the buffer and then the real start of message comes and will put in as byte 2 in the buffer, this will of course fail the checksum but I don't want to end up in a situation where no message comes through as the receive buffer is out of sync as it is picking up any garbage.

Yes 0x55 followed by 0xAA will be more robust, the longer the better until the starter bytes exceed the actual useful data 

[errorprone]

Can always use a standard such as Modbus RTU where frames are delimited by a silent period of 3.5 characters or Modbus ASCII which always starts with a colon and ends with a carriage return and line feed.  Both are typical over RS485 and there are plenty of libraries available so it doesn’t need to be written from scratch.

[Simon]

well standards need taking as are so I might take on more than I need. The basic problem is solved by having a unique enough start sequence which as you say modbus ASCII has with the penalty of now having to convert everything into characters. The idea of a fixed time between frames does not sound as good, you have to start at some point so will throw the first message away if there is some gibberish followed by a message soon after.

When you say that there are libraries, where would I find these and what form do they take? This is a problem that is very tied to setting up the specific hardware.

[langwadt]

I think you are over thinking this, if there's garbage on the line you'll miss frames, there's no way around it, and since RS485 is a shared bus any spurious data will affect everyone on the line and the communication must have some form of master requesting responses from each slave at a time

[Simon]

no not necessarily. I don't know what it takes for the micro controller to detect a valid byte. I guess it will measure from the start to the end of the frame and only take the byte as good if it sees the correct timing. Now if for some reason it "sees" a 0 byte on the bus it will put that into the first byte of the buffer variable without questioning it. If a message then starts this will continue to fill the buffer. The message is now one byte out and the checksum fails.

Now I don't know what happens with the RS485 transceiver as now TX and RX are joined. the flow control pin will disable the input when the output is active, I have thought maybe it is a good idea to disable the receiver while D transmit.

[peter-h]

If you control the protocol then it is fairly easy.

Define a packet which is all ASCII, starts with say "*" and ends with say "$".

Throw away bytes until you get a * and then load them into a buffer (bigger than needed), with a timeout of say 1 sec. If the timeout goes, go back to start. If end of buffer reached, go back to start. If $ seen and there was no timeout and you are not at the end of buffer, check a CRC or some such, which was last 2 bytes (four ascii characters) before the $.

With RS485 you are always master-slave. The master polls and the slave responds. If your application is full duplex then you can't use RS485. You have to use RS232 or RS422 (RS422 is often called 4-wire RS485).

Yes you enable the RS485 driver during transmission, then turn it off. This can be tricky; normally you transmit under interrupt, and when you get the TX Empty ("all sent") interrupt you disable the driver. The driver is enabled when data is loaded into the UART.

I would not bother with Modbus. Over-complicated, and using some library just introduces code which you don't know how it works.

[Siwastaja]

Yes the plan is a message format with an address and error check at the end. My concern is that if a reception is triggered by anything I could never be in sync.

Consider this usual stream format:
START LEN PAYLOAD CRC

By choosing START magic number properly, odds that it is repeated in payload are made smaller (can be multibyte):
acdcabba 05 (01 02 03 04 05) 13 37

Now it's still possible that acdcabba repeats in data:
acdcabba 05 (01 ac dc ab ba) 21 18

When you start parsing at correct start id, CRC check succeeds -> you are now in sync with the stream.

But let's say you are out of sync and accidentally sync to the wrong acdcabba. Now 0x21 in above example is interpreted as length. You have to read many bytes and lose many packets before you get to what you think is CRC, get mismatch and only then detect the error.

Worse, if the unlucky packets containing acdcabba are recurring, chances are you always try to resynchronize on the wrong start id, and never get a sync.

Solution to this is to buffer bytes and if CRC mismatch happens, scan old bytes in the buffer to find another acdcabba. One of them must be the right identifier. You can guarantee to get back in sync this way, but scanning the whole buffer many times is more work and also requires much more processing than byte-by-byte parsing while bytes are received.

Or, you can just accept not to cover this problem. With a long (say 32-bit), randomly chosen magic number and low error rates, the failure condition of never getting sync is extremely rare.

One way of mitigating the problem of having to waste a lot of packets due to misidentification or corruption of length field, is to use separate CRC for header only. For example, a 8-byte header block: START32 LEN16 MSGID8 CRC8, followed by payload and then CRC16 over the payload. Now, chances of this 64-bit sequence (including valid CRC) occurring in payload is extremely small, and that repeating in every data frame even more slim.

This post is assuming you want to use all possible byte values in payload. If you are willing to remove 1:1 mapping from raw payload to packet payload, you can choose unique values for start/end identifiers as peter-h suggests, but then you need to escape these values in data, and data length becomes different to original raw payload.

ASCII is good when you want human interoperability, but on the other hand, ASCII-based systems have been notoriously troublesome, despite good intentions.

[peter-h]

The problem with binary data is that you must rely on timeouts to handle error conditions, and/or implement weird speculative approaches (like speculatively checking the CRC as you go along).

Whereas with ascii data (perhaps just straight hex) it is all very easy. Sure, the packets are 2x longer but usually this doesn't matter.

[pcprogrammer]

When using 7 bit ASCII the msb is available to be used in the preamble. MIDI uses this. When the msb is one it indicates a command byte. So use for instance 0xA5 0x96 as the start of the message and only 7 bit ASCII for the rest. Limit your payload size to 127 bytes and code your CRC in ASCII HEX.

Adds a bit to the length and processing of the message, but the original post mentions human machine interface, and how much data at what rate can that mean.

[Simon]

using ASCII does feel like it's a pain, I have 8, 16 and 32 bit numbers to send and bit flags. my thinking is:

1) Timing out allows me to be sure any ongoing message has ended
2) with the certainty that any new byte is a new message it must be "the one", using 2 bytes to start gives me a 1 in 65536 chance of failure instead of 1 in 256. This sounds pretty good, I don't know if having 1 start byte and a second start byte that relates to the first further improves on this like they should both add up to 0xFF so the second is also a checksum of the first as will as making it a 16 bit start instead of 8 bit.
3) having information about the data length could be useful to further be more sure about the message end. I could combine this with the start byte maybe, either a variable start sequence that much pass a checksum or just have a start, message length and then checksum of those 2, that now means that there is less than 1 in 16 million chance that there is a misinterpretation.
4) A stop byte would make matters robust as although it could repeat in the data it would have to be in that precise location it is expected and with a message length either predetermined or given at the start with checksum protection very robust.

I think essentially anything would be way better than no start identifier. Making sure it's something better than just a start byte will be fine.

Even if I were to add several bytes of overhead this would still beat the extra data from ASCII and would avoid converting the data at each end. Even if I did want to send text around the basic alphabet upper and lower case with the numbers 0-9 fit into 64 bits so you could send a really robust header and encode the text with 3-4 characters per byte. But I am not planning to send text.

[Simon]

When using 7 bit ASCII the msb is available to be used in the preamble. MIDI uses this. When the msb is one it indicates a command byte. So use for instance 0xA5 0x96 as the start of the message and only 7 bit ASCII for the rest. Limit your payload size to 127 bytes and code your CRC in ASCII HEX.

Adds a bit to the length and processing of the message, but the original post mentions human machine interface, and how much data at what rate can that mean.

Right now I am doing something in a hurry with 2 devices so as long as it is sensible enough to not fail in front of a customer on a demo it's fine. In the longer term I will have multiple devices so I'm trying to make sure I'm not doing something that is just a fix but carries a lot of overhead that later ends up being a pain to live with.

[langwadt]

There's also a lot of uarts that directly support using the parity as a ninth bit

[Siwastaja]

Whereas with ascii data (perhaps just straight hex) it is all very easy. Sure, the packets are 2x longer but usually this doesn't matter.

What, you don't implement error checking such as CRC on ASCII? Of course it is "easier" if you set different goals.

ASCII parsers tend to be at least 10 times more complex than binary parsers. If you are willing to pay this price for human interoperability, this is fine, and a good tradeoff. But if you use ASCII thinking it is "simpler" to program (and works more robustly), you are terribly misled.

[Simon]

A 4 byte number could be as long as 10 characters, a 2 byte number could be as long as 5 characters, a single byte number could be as long as 3 characters, ASCII is not the way to go, while it may guarantee that a single start byte will be unequivocally recognized a 3-4 byte starter that would contain some helpful information like length would be far less overhead.

[Siwastaja]

One middle-ground option which gives you some level of human interoperability, leaves unused symbols for start/end delimiters, but is less wasteful on comms bandwidth, and less wasteful on CPU number conversion cycles, and what bests, generates fixed-length datagrams from binary data, is ASCII but with hexadecimal numbers instead of decimal. Being constant-length, you can theoretically leave out field delimiters, then it "only" doubles the size compared to binary.

Did I get the premise correctly, all participants are microcontrollers, no PC stuff? If so, to get something quickly, I would start by implementing the packet framing using idle time, as suggested by many. Then just send binary data as is. No need to think about CRCs for the first prototype. Instead, concentrate on RS485 signal integrity; you have to do that anyway! If you do that, bit errors will be rare enough you can implement error checking later.

Remember that RS485 needs biasing (misleadingly called "failsafe biasing"; it has nothing to do with failing safely) to operate correctly; look it up. Some transceiver ICs are fine without, some are marginal, some don't work reliably at all.

[Simon]

The actual communication seems to be fine. but when I tried receiving data from the other device (ESP32) and then sending it back I was sending back 0's. But there was also some data that changed. So I have become suspicious that I am getting out of sync with the datastream as I have no way of knowing that the byte I just received really is valid.

[SuzyC]

Do what some TV Remote Control codes formats do.

Very high security esp. for long frame: Send each byte in your frame and then the frame as complement ~bytes and compare each data frame byte to validate frames. This works very well if you have long frames and the time available to put up with the overhead.

Even better is adding to the above a one to 4-bytes fixed preamble and a single fixed valued end byte after an 8-bit CRC.  This protocol can be shortened to apply only to the preamble bytes, adding a 8-bit CRC at the end, if more speed and less security is desired.

The write enable protocol used by some EEPROMs is also very reliable, using a fixed 4-byte preamble followed by the data frame and and a fixed ending byte preceded by a 16-bit CRC.

The possibilities are endless, if you have the time.

[Siwastaja]

The actual communication seems to be fine. but when I tried receiving data from the other device (ESP32) and then sending it back I was sending back 0's. But there was also some data that changed. So I have become suspicious that I am getting out of sync with the datastream as I have no way of knowing that the byte I just received really is valid.

Well, you have to debug your system, look at the signals carefully with scope, save and print intermediate values, compare, whatever is needed. You can toggle extra IO pins from your code so that you can see that something happened in synchronization with the RS485 data on scope.

Any communication strategy is going to require some actual work and unless you are a mega genius, nothing will just work out of box. Try to work out what's wrong with your current code first.

[Simon]

Any communication strategy is going to require some actual work and unless you are a mega genius, nothing will just work out of box. Try to work out what's wrong with your current code first.

Indeed. And the first task in a protocol receiving data is to ascertain that what was received is the start of a message, looking at the whole message for validation and not just relying on a byte appearing in the buffer with no further checks.

[pcprogrammer]

... unless you are a mega genius, nothing will just work out of box. Try to work out what's wrong with your current code first.

And that is the real skill of programming.  Being able to figure out why something that in your mind should work is not working.

[Simon]

the datasheet (SAMC21) states:

31.6.2.6 Data Reception
The receiver accepts data when a valid start bit is detected. Each bit following the start bit will be sampled according
to the baud rate or XCK clock, and shifted into the receive shift register until the first stop bit of a frame is received.
The second stop bit will be ignored by the receiver.
When the first stop bit is received and a complete serial frame is present in the receive shift register, the contents
of the shift register will be moved into the two-level receive buffer. Then, the Receive Complete interrupt flag in the
Interrupt Flag Status and Clear register (INTFLAG.RXC) will be set, and the optional interrupt will be generated.
The received data can be read from the DATA register when the Receive Complete interrupt flag is set.

I'm not entirely clear here. If I read this literally the buffer will just keep accepting the "data" until the line goes high, so if there is a spurious low pulse I will get a "0" byte if it lasts longer than 9 bits worth.

[pcprogrammer]

That is how a UART works. A start bit is when the line goes low and stays low during a bit time. Then the shift register gets filled with what it sees on the line. If that is zero for the duration of full data set, you will see a zero.

A modern UART uses oversampling and other techniques to reduce error, but it still boils down to seeing a start bit and then clock the data into a shift register. If the stop bit is not seen at the right time it can raise a framing error, but you have to check for it. Think it will still signal a data set received.

To get an insight you could read this article about an implementation in VHDL or verilog.

[paulca]

On minimal, robust, small footprint, performance messaging protocols for text streams.  It could be worth spending 5 minutes scanning the FIX protocol.

https://www.fixtrading.org/

For the context here, you can ignore all the application tags and only consider the protocol tags.

Features of use, probably on UART.

* Preamble w/ message type and length.
* Postamble with checksum and trailer
* Bi-directional message sequence numbering
* Reconnection with or without correct sequence numbers*
* Message resend or "gap fill" to resynchronize.  (in the case of an MCU I would expect that to be 99.9% "gap fill" only maybe keeping messages classed as "Last will and testament" priority.
* bidirectional heartbeats/pings

*connection in this case refers to the handshake of establishing sequence numbers and "identity" of the party.  While TCP is usually used for the protocol, FIX can arguably be sent on any async bi-directional protocol.

EDIT: I am going to try and use it for my own purposes.  A few notes:

It should not be hard at all to find an extremely high performance, minimal footprint FIX parser in C or ASM for Arm.  (There I would need to be a little careful with latent IP).  The basic parser works much like any UART protocol of "mixed length" async messages.  Read the minimal header to detect an incoming message, start streaming it into a tokenizer which pops out tag=value pairs at you.  Keep popping them onto a stack until you get the length.  Then you can reserve space and let the stream run while you carry on tokenizing your way through the message.  You can tokenize it into a big tree structure in memory which is slow, but only needs done once, or you can "strike through" process and trigger events in a state machine as and when combinations of tags are received without waiting on the rest of the message.  Risky, but where this is used any dirty trick makes money.

In the real world a lot of code goes into the session layer with identities and parties and authentication/encryptions, but we don't need none of that non-sense here.  So other than checking the SenderCompID is "Your devices's ID", all of that goes away.  Sequence number coherence is up to the sender and receiver.  In the most simple form, you just sent gap fills both directions or demand a reset sequence numbers on all connections and abandon resent completely.  It can be very, very handy though, if you get SQ# 1, 2, 3, 6, 7... you can send a "WTF?" message back asking for 4 and 5.  On low foot print without memory to reorder yourself, you could just pretend you didn't get 6 or 7 either.  Given the other end might be low memory as well, it's likely to just say, "Sod it, new sequence number is 8, we are good.".  It's just that the protocol has mechanisms to enforce these so that both parties are 100% aware of any message loss.

I think one hurdle in your case will be the ModBus/RS485 protocol of any other devices on the bus.  You will need to at minimum filter only messages from devices you want at the Modbus layer.  Also.  Doesn't modbus work more like I2C with register sequence read/write.  You could wrap that in FIX to get resilience, convert registers to tags or just a blob in one tag.  FIX might be better suited to bar UART.

[woofy]

the datasheet (SAMC21) states:

31.6.2.6 Data Reception
The receiver accepts data when a valid start bit is detected. Each bit following the start bit will be sampled according
to the baud rate or XCK clock, and shifted into the receive shift register until the first stop bit of a frame is received.
The second stop bit will be ignored by the receiver.
When the first stop bit is received and a complete serial frame is present in the receive shift register, the contents
of the shift register will be moved into the two-level receive buffer. Then, the Receive Complete interrupt flag in the
Interrupt Flag Status and Clear register (INTFLAG.RXC) will be set, and the optional interrupt will be generated.
The received data can be read from the DATA register when the Receive Complete interrupt flag is set.

I'm not entirely clear here. If I read this literally the buffer will just keep accepting the "data" until the line goes high, so if there is a spurious low pulse I will get a "0" byte if it lasts longer than 9 bits worth.

Yes, you will get a spurious byte. Most UART's use oversampling, which in practice this means they take 3 samples in the middle of the bit time and use majority voting to decide the bit. The thing about real world interference is that it is very short sharp spikes, not continuous highs or lows. Unless you are using very high baud rates the UART can filter a lot of this out for you.

If you have control of the CPU in the HMI and an available uS level timer then the scheme I outlined at the start of this thread should provide a simple but robust solution. It's based on one of my own applications (my day job is designing industrial controls)  where I use 250k baud RS485 communication on anything up to 1km as it routes through the building(s), with up to 128 devices on the bus. I use a leading zero byte, followed by the target address, payload length, payload and CRC. Using 9-bit data gives a 44uS byte time and bytes are always transmitted back to back. I set/reset a 60uS timer for each byte and if it times out before the next byte is received then that's end of frame. I can then verify the length, verify the CRC and extract the payload.
On the other hand if you are working through an OS in the HMI then this may not be possible.

[Simon]

A modern UART uses oversampling and other techniques to reduce error, but it still boils down to seeing a start bit and then clock the data into a shift register. If the stop bit is not seen at the right time it can raise a framing error, but you have to check for it. Think it will still signal a data set received.

This is what I fear the issue is, and with the RX and TX tied together there is a chance of getting such gibberish.

[pcprogrammer]

That is the thing with RS485. At the sending terminal you will receive what you send. If there is the need for some multi master setup you need collision detect and bus arbitration. But in a simple setup as laid out in the original post it seems like just one master (HMI) and one slave (ECU), at least for your first tests, all you need to take care of is that the slave (ECU) never enables it's transmitter unless asked to do so by the master (HMI). The master needs to disable his transmitter to allow the slave to respond.

So to report status, you need to poll it, initiated by the master.

If you have problems with receiving wrong data in your test case, I would strongly advice, like already given by someone else, to hook up some test equipment on the bus and check what is happening on it.

[PCB.Wiz]

The actual communication seems to be fine. but when I tried receiving data from the other device (ESP32) and then sending it back I was sending back 0's. But there was also some data that changed. So I have become suspicious that I am getting out of sync with the datastream as I have no way of knowing that the byte I just received really is valid.

Try single byte echoes first, with something trivial like add-one or invert to the echo. That does not even need interrupts and confirms HW and Baud are ok.
Then you can step up to a burst of eg incrementing numbers, and check your buffers and interrupts do not overflow.

Using a gap as a block identifier is great, provided you have systems that do not have complex operating systems that can create a gap
Typically a timer-monostable is reloaded on every Char RX-INT, and when the timer actually times out, you know you had a gap. Set that to maybe 80% of the actual time so yo u have room to check size/checksums etc

This is what I fear the issue is, and with the RX and TX tied together there is a chance of getting such gibberish.

They should be valid whole characters, and during TX you usually accept and discard RX, tho you could confirm they matched TX.
Some RS485 designs pre-bias the cable, so that a floating driver does not sit close to the receive threshold.

[Simon]

i have a SAMC micro controller and an ESP32 (colleagues work), me sending works. Me receiving not so. I'll have a go tomorrow with the start byte, I suspect that the transmission is creating a ghost reception so when I do come to receive I am one byte into the buffer already, I send that back and slowly it all turns to 0's as at each message a byte is dropped for a "0".

[woofy]

i have a SAMC micro controller and an ESP32 (colleagues work), me sending works. Me receiving not so. I'll have a go tomorrow with the start byte, I suspect that the transmission is creating a ghost reception so when I do come to receive I am one byte into the buffer already, I send that back and slowly it all turns to 0's as at each message a byte is dropped for a "0".

What are you using for the RS485 transceiver? The usual device is an 8-pin ST485 or equiv. These have separate drive and receive enables, DE active high and RE active low. Just tie them together and you cannot receive what you transmit. You will need a pullup on the RO as RE tristates it. If you have them tied together, but no pullup resistor, that could be the source of your spurious byte.

[Simon]

Yes I was wondering about that, I think it is the falling Rx line when I transmit that was triggering a reception.

It's all working now with a start byte.