Author Topic: How should be a serial communication packet format ?  (Read 1411 times)

packetbob and 1 Guest are viewing this topic.

Offline Kjelt

  • Super Contributor
  • ***
  • Posts: 3601
  • Country: nl
Re: How should be a serial communication packet format ?
« Reply #25 on: July 17, 2017, 03:49:13 AM »
I also add an <end of packet> byte for robustness and a timegap between two packets >2 bytes Timing.
So if the lenght exceeds the lenght byte or the time that a byte was received exceeds one and half byte, i drop the packet and wait for a new header. Since I use acknowledgments on each transfer the master knows if the transmission failed.
« Last Edit: July 17, 2017, 03:51:11 AM by Kjelt »
 

Offline Delta

  • Super Contributor
  • ***
  • Posts: 1114
  • Country: gb
Re: How should be a serial communication packet format ?
« Reply #26 on: July 18, 2017, 02:13:24 AM »
With only 26 bytes to send each second, just keep blootering the same packet over the link. If your receiver gets 1 packet differing from the other 23, it isn't rocket surgery to work out what is good data and what is noise.
 

Offline Pack34

  • Frequent Contributor
  • **
  • Posts: 460
Re: How should be a serial communication packet format ?
« Reply #27 on: July 18, 2017, 04:26:03 AM »
Will you be sending multiple types of packets at different times? Or will you be streaming a simple and standard control message? You could simply send

START BYTE (0xFF)
VAL1
VAL2
VAL3
...
VALN
CHECKSUM
END BYTE (0xFE)

Then just be sure all of the payload and checkum bytes are properly masked such that your START and END bytes cannot appear. This way you simply spool until you see 0xFF appear on the bus, read until you see 0xFE, validate the checksum, then parse your internal bytes.

Quick and dirty.
 

Offline Kjelt

  • Super Contributor
  • ***
  • Posts: 3601
  • Country: nl
Re: How should be a serial communication packet format ?
« Reply #28 on: July 18, 2017, 04:35:09 AM »
That makes no sense, when transferring data all 255 values could be valid. Just add a length field(byte) after the start byte and the protocol is robust enough.
Already use something similar for over twenty years 24/7 works like a charm, at least if you can write code.
 

Online Kalvin

  • Super Contributor
  • ***
  • Posts: 1337
  • Country: fi
Re: How should be a serial communication packet format ?
« Reply #29 on: July 18, 2017, 04:35:50 AM »
I typically prefer to send the length field as non-inverted and inverted. That will provide a fast way to detect whether the packet length is valid or not. There may be some better ways to improve the error detection of the length field by using some clever coding, something similar to what hamster_nz suggested in his earlier post http://www.eevblog.com/forum/microcontrollers/how-should-be-a-serial-communication-packet-format/msg1257528/#msg1257528.
 

Offline Hensingler

  • Regular Contributor
  • *
  • Posts: 144
  • Country: gb
Re: How should be a serial communication packet format ?
« Reply #30 on: July 18, 2017, 05:24:02 AM »
That makes no sense, when transferring data all 255 values could be valid. Just add a length field(byte) after the start byte and the protocol is robust enough.

Any protocol depending on a length field is a fail. You don't know you have a valid length field without checking the CRC and you can't check the CRC without a valid length field. A length field corrupted long means you might read in the next several packets before checking something that isn't a CRC.

So you then have to start adding time outs and the associated real time requirements for the transmitter (like you already posted you did) because you chose dumb protocol to start with.
 

Online nctnico

  • Super Contributor
  • ***
  • Posts: 11981
  • Country: nl
    • NCT Developments
Re: How should be a serial communication packet format ?
« Reply #31 on: July 18, 2017, 05:35:01 AM »
That makes no sense, when transferring data all 255 values could be valid. Just add a length field(byte) after the start byte and the protocol is robust enough.
Any protocol depending on a length field is a fail. You don't know you have a valid length field without checking the CRC and you can't check the CRC without a valid length field. A length field corrupted long means you might read in the next several packets before checking something that isn't a CRC.

So you then have to start adding time outs and the associated real time requirements for the transmitter (like you already posted you did) because you chose dumb protocol to start with.
That is only true if the transmitter sends packets before it receives a reply (at least an acknowledge). So far the only protocol I have come across which does that is TCP/IP but that sits on top of a link layer which already does length and CRC checking.
There are small lies, big lies and then there is what is on the screen of your oscilloscope.
 

Offline Kjelt

  • Super Contributor
  • ***
  • Posts: 3601
  • Country: nl
Re: How should be a serial communication packet format ?
« Reply #32 on: July 18, 2017, 05:39:20 AM »
Quote
you chose a dumb protocol
Pffff i like the KISS principle, if it works, it works. If it fails even after a year I analyze and change. If youwant robustness to the max
then create a header with its own checksum if you are so paranoid like the ip protocol and be done with it. My system is wired rs485 and have not had a single fail in 20 years with my dumb protocol. Design your own and be happy, i am happy with my choice.
 

Offline nfmax

  • Frequent Contributor
  • **
  • Posts: 516
  • Country: gb
Re: How should be a serial communication packet format ?
« Reply #33 on: July 18, 2017, 05:46:30 AM »
That makes no sense, when transferring data all 255 values could be valid. Just add a length field(byte) after the start byte and the protocol is robust enough.

Any protocol depending on a length field is a fail. You don't know you have a valid length field without checking the CRC and you can't check the CRC without a valid length field. A length field corrupted long means you might read in the next several packets before checking something that isn't a CRC.

Real communications protocols get round this using a fixed-length header containing the packet size field, with a header checksum.

The OP is better off using an already existing protocol, rather than trying to invent one just for this purpose, even if the existing protocol is massive overkill for the job. In another galaxy, long, long ago, we did just this. Used the Kermit protocol between a hand-held controller and a data acquisition system. This had to operate in the carrier flight deck environment, 200V/m EMF, so we needed good error correction. Nobody needed to know what the protocol was (we made both boxes) so we just kept schtumm about it!
 

Offline Hensingler

  • Regular Contributor
  • *
  • Posts: 144
  • Country: gb
Re: How should be a serial communication packet format ?
« Reply #34 on: July 18, 2017, 05:47:56 AM »
That is only true if the transmitter sends packets before it receives a reply (at least an acknowledge).

If the transmitter doesn't send more packets the receiver will wait for ever to receive the corrupted long length making the time outs mandatory. There no requirement for replies anyway. I have implemented several unidirectional serial links and networks where only commands which queried information produced response messages, in particular broadcast messages didn't.

Real communications protocols get round this using a fixed-length header containing the packet size field, with a header checksum.

Which is just pointless overhead and less reliable than using unique message separator or start/stop symbols. If you get a break in the comms link (quite common for radio) you will still be sitting there waiting for your validated length message that will never come (and make time outs a requirement).

 
« Last Edit: July 18, 2017, 05:55:54 AM by Hensingler »
 

Offline ejeffrey

  • Super Contributor
  • ***
  • Posts: 1382
  • Country: us
Re: How should be a serial communication packet format ?
« Reply #35 on: July 18, 2017, 06:11:18 AM »
Quote
you chose a dumb protocol
Pffff i like the KISS principle, if it works, it works. If it fails even after a year I analyze and change. If youwant robustness to the max
then create a header with its own checksum if you are so paranoid like the ip protocol and be done with it. My system is wired rs485 and have not had a single fail in 20 years with my dumb protocol. Design your own and be happy, i am happy with my choice.

The simple thing is to use one of the existing protocols that already solves this, not rolling your own either way.  And honestly, I don't believe that you have never had a problem if you really did what you describe.

Serial ports can and will generate garbage bytes when the microcontroller resets.  The user will restart the host application in the middle of a message.  If you have a serial cable with a connector, someone will disconnect and reconnect the port.  The problem with using length bytes for framing is that if you loose your frame pointer, you are lost forever until you restart the whole system.  So I don't believe that you have used this protocol with out any way to recover synchronization.  Your way is probably just not as good as a protocol that auto synchronizes every message.  There is no need to use a protocol that doesn't support this.  It is trivial to dedicate a character to frame boundaries and then escape it when it occurs in data.  If you really care about the worst case overhead where every byte in a message has to be escaped, and you can't just increase your line rate, then use a slightly more complicated protocol.

All more modern / higher level communications interfaces have built in ways to do this.  Even GPIB, which is nearly as ancient as RS232 has this with the EOI line.  Modern protocols like ethernet, USB, etc. all have built-in message framing.  Hell, even async serial actually has this capability, you can generate a BREAK to synchronize the sender and receiver, except that handling of BREAK condition is pretty hit or miss between implementations.
 

Offline Kjelt

  • Super Contributor
  • ***
  • Posts: 3601
  • Country: nl
Re: How should be a serial communication packet format ?
« Reply #36 on: July 18, 2017, 06:27:24 AM »
Too long to type here on an ipad in hotel but no i have no problems since i do have synchronization start stop frame bytes length, command and crc. I have 30 slaves and one master , can do individual control as well as broadcast and it works, if it did not i would notice since they are the lights in my livingroom and other things as blends for screens. The reason i definitely wanted a length field is to accomodate new additions that might require a longer or shorter messages and i wanted future expansion  flexibility without having to reflash all firmwares from other slaves.

I agree that whoever wants to do this should re-use existing protocols, however for me it was part of the fun to design my own and basically i found many other similar protocols from others.
If you only copy that which others have made then you will not learn new things.
One of the protocols i came across was a fixed length protocol. So frames were always the same size either too long or too short and inflexible as you can think of. If that fits your design go do it. If you want to reserve some of the bytevalues for your protocol do it, i don't want that, if I sniff my protocol i can read it , i don't have to decode or recalculate it. If i want to encrypt my payload i can do it, i don't have to worry that some bytevalues are reserved.
 

Online mikeselectricstuff

  • Super Contributor
  • ***
  • Posts: 9670
  • Country: gb
    • Mike's Electric Stuff
Re: How should be a serial communication packet format ?
« Reply #37 on: July 18, 2017, 06:27:43 AM »
  Hell, even async serial actually has this capability, you can generate a BREAK to synchronize the sender and receiver, except that handling of BREAK condition is pretty hit or miss between implementations.
And can get even more interesting when the receive UART has a FIFO. I don't recall ever seeing an MCU datasheet that clearly explained exactly how its FIFO'd UART deals with break conditions.
The other problem with out-of-band features like breaks is it gets messy when somewhere down the line you need to transport the UART stream over another protocol like USB or ethernet.
Youtube channel:Taking wierd stuff apart. Very apart.
Mike's Electric Stuff: High voltage, vintage electronics etc.
Day Job: Mostly LEDs
 

Offline mikerj

  • Super Contributor
  • ***
  • Posts: 1256
  • Country: gb
Re: How should be a serial communication packet format ?
« Reply #38 on: July 18, 2017, 07:42:29 AM »
Which is just pointless overhead and less reliable than using unique message separator or start/stop symbols. If you get a break in the comms link (quite common for radio) you will still be sitting there waiting for your validated length message that will never come (and make time outs a requirement).

If your unique message separator gets corrupted, what stops the receiver waiting for the end of of a packet which it's already missed, if timeouts are not an option?
 

Offline Naguissa

  • Contributor
  • Posts: 5
  • Country: 00
    • Foro de electricidad, electrónica y DIY / HUM en español
Re: How should be a serial communication packet format ?
« Reply #39 on: July 18, 2017, 08:07:47 AM »
Header - length - base64(data) - crc (also base64) - end mark.

Header and end should be non-base64 symbols.

Edit: one note, length is for 2nd level error checking, not for framing.

Enviado desde mi Jolla mediante Tapatalk
« Last Edit: July 18, 2017, 08:17:03 AM by Naguissa »
 

Offline Hensingler

  • Regular Contributor
  • *
  • Posts: 144
  • Country: gb
Re: How should be a serial communication packet format ?
« Reply #40 on: July 18, 2017, 08:28:59 AM »
If your unique message separator gets corrupted, what stops the receiver waiting for the end of of a packet which it's already missed, if timeouts are not an option?

It is waiting, it didn't receive a corrupt packet so what. At all times receiving a start symbol restarts collection of a packet. The start/stop or separator scheme guarantees that a received sequence of bytes forming a valid packet will *always* be interpreted as a valid packet regardless previous state. Something which protocols with length fields and time outs do not.

As an aside you can not make any assumption about a sequence of bytes which is not a valid checksum verified packet - it can not be relied on to have more meaning than received noise or received nothing. If you need to look at it any of it before verification your protocol is broken.

Header - length - base64(data) - crc (also base64) - end mark.

Settle for a constant 25% base64 encoding overhead while a 3 'special' symbol start/stop/quote scheme gives a variable 1.2% average overhead and much simpler implementation.
« Last Edit: July 18, 2017, 08:35:24 AM by Hensingler »
 

Offline krho

  • Regular Contributor
  • *
  • Posts: 99
  • Country: si
Re: How should be a serial communication packet format ?
« Reply #41 on: July 18, 2017, 03:16:59 PM »
Settle for a constant 25% base64 encoding overhead while a 3 'special' symbol start/stop/quote scheme gives a variable 1.2% average overhead and much simpler implementation.
  :-+

Or even better
 

Offline Jeroen3

  • Super Contributor
  • ***
  • Posts: 2254
  • Country: nl
  • Embedded Engineer
    • jeroen3.nl
Re: How should be a serial communication packet format ?
« Reply #42 on: July 18, 2017, 03:46:08 PM »
Put it in 9 bits mode. Enable 9th bit on control characters.
 

Offline Karel

  • Frequent Contributor
  • **
  • Posts: 934
  • Country: 00
Re: How should be a serial communication packet format ?
« Reply #43 on: July 18, 2017, 04:32:31 PM »
And the winner is.....

https://www.rfc-editor.org/rfc/rfc1055.txt

https://en.wikipedia.org/wiki/Serial_Line_Internet_Protocol

Simple, efficient and elegant, including sourcecode.
The difference between theory and practice is less in theory than
the difference between theory and practice in practice.
Expensive tools cannot compensate for lack of experience.
 

Offline voltsandjolts

  • Supporter
  • ****
  • Posts: 338
  • Country: gb
Re: How should be a serial communication packet format ?
« Reply #44 on: July 18, 2017, 05:24:57 PM »
And the winner is.....

https://en.wikipedia.org/wiki/Consistent_Overhead_Byte_Stuffing

http://stuartcheshire.org/papers/COBSforToN.pdf

Simple, very efficient and elegant, including sourcecode.
 

Offline Jeroen3

  • Super Contributor
  • ***
  • Posts: 2254
  • Country: nl
  • Embedded Engineer
    • jeroen3.nl
Re: How should be a serial communication packet format ?
« Reply #45 on: July 18, 2017, 05:29:03 PM »
Maybe we should make a better one?
 

Offline veryevil

  • Supporter
  • ****
  • Posts: 143
Re: How should be a serial communication packet format ?
« Reply #46 on: July 18, 2017, 07:46:15 PM »
I'd like to put in another vote for SLIP. used it many times in the past. Easy to encode. Easy to decode. Any length you want.

Code: [Select]
#define SLIP_END 0xC0
#define SLIP_ESC 0xDB
#define ESC_END 0xDC
#define ESC_ESC 0xDD


void EncodeSlip(char *buffer_ptr, unsigned int buffer_length_bytes)
{
unsigned int i = 0;
printf("%c", SLIP_END);            // Write start of SLIP frame   

while (i < (buffer_length_bytes)) {          // Send all buffer in SLIP format   
switch (buffer_ptr[i]) {                   // check to see if is a special character   
case SLIP_END:   
printf("%c%c", SLIP_ESC, ESC_END);     // escape special character   
break;   

case SLIP_ESC:   
printf("%c%c", SLIP_ESC, ESC_ESC);     // escape special character   
break;   

default:   
printf("%c", buffer_ptr[i]);        // send raw character   
}   
i++;   // continue with next character send   
}   
printf ("%c", SLIP_END);
}

 

Offline voltsandjolts

  • Supporter
  • ****
  • Posts: 338
  • Country: gb
Re: How should be a serial communication packet format ?
« Reply #47 on: July 18, 2017, 08:01:21 PM »
SLIP is nice and simple. However, it can create a lot of overhead (potentially double the number of bytes to transfer) if you are sending a lot of bytes which happen to be the same as SLIP_END. If you need to maintain transfer rates with minimal overhead regardless of data content, then use COBS (links in my post above).
 

Offline Karel

  • Frequent Contributor
  • **
  • Posts: 934
  • Country: 00
Re: How should be a serial communication packet format ?
« Reply #48 on: July 18, 2017, 08:09:11 PM »
When using SLIP, it can be convenient to change the SLIP_END character to 0x0A (newline).
This is handy for testing and sending commands manually via a terminal (no need to encode the commands).
The difference between theory and practice is less in theory than
the difference between theory and practice in practice.
Expensive tools cannot compensate for lack of experience.
 

Offline andyturk

  • Frequent Contributor
  • **
  • Posts: 653
  • Country: us
Re: How should be a serial communication packet format ?
« Reply #49 on: July 18, 2017, 10:06:27 PM »
It is trivial to dedicate a character to frame boundaries and then escape it when it occurs in data.  If you really care about the worst case overhead where every byte in a message has to be escaped, and you can't just increase your line rate, then use a slightly more complicated protocol.

Bingo.

COBS allows you to unambiguously dedicate a single octet (byte) to framing. The sender encodes every message (whatever the length) using COBS and appends the framing octet (e.g., 0x00) to the end. The receiver reads everything up to a 0x00 into a buffer and reverses the COBS encoding to recover the original message.

Essentially what COBS does is guarantee that 0x00 *never* occurs in encoded form (with only space 0.4% overhead). That's how you can use 0x00 on the receiving side to delimit your frames. COBS deals with frames of arbitrary length and gives you the option of dispensing with an explicit length within the frame itself.

More info here: https://www.embeddedrelated.com/showarticle/113.php

EDIT: I see voltsandjolts is pounding the same drum.  :-+
« Last Edit: July 18, 2017, 10:08:48 PM by andyturk »
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf