How should be a serial communication packet format ?

[sangar]

Hello all,

I have to transmit 26 bytes of payload for every second from PIC18 to PC through Serial communication. Right now , I don't enable parity bit any thing. I frame the payload like below.


Header(1-byte)     Length(1-byte)   Payload      Footer(2-byte).


Is the above safe?


As I am using Low end Micro-controller (PIC18), I don't want to implement CRC. Suggest if anything which works better.



Thanks,
Sangar 

[krho]

If you don't want to implement the CRC because of space constraint then you can implement fletcher16. This one is simple and small

[sangar]

Hi,
Thanks.


Fletcher-16  will give 2 bytes of CRC ?

I want to produce 1-byte CRC for 26 bytes . Can you share sample code?

[Rerouter]

It comes down to if every packet being accurate matters to you, if you want to catch a certain amount of errors, you can use a parity bit or an xor check,

[voltsandjolts]

This might help you choose:
https://users.ece.cmu.edu/~koopman/pubs/KoopmanCRCWebinar9May2012.pdf

[cv007]

or just send it twice- if 2 'packets' are not identical, then there is a problem. 26 extra bytes does not take a lot of time to send and you already have the code to send it once, so just send it twice.  208 bits of parity

[Jeroen3]

How will you ensure synchronisation? What if the header/footer appear in the payload?
With this protocol, you will have use some timeout mechanism to indicate a new packet is arriving.
Eg, what if something goes wrong?:
header - payload - footer ...... header - pa.. [interruption] hdr - payload - footer ...

I have made a reader for a device that used this kind of protocol. Packets would start with 0xAA, some payload of variable length, and end with a CRC.
It was impossible to keep it synchronized, the pc won't give nice full packets in the received events.
I ended up to try to decode a packet and check crc each 0xAA that appeared in the stream. An awfully inefficient method.

When working with streams, I prefer to use ascii since you can use dedicated flow control characters. It makes the implementation easier and more robust, at the cost of troughput.
Since you're going to the PC, at a very low rate, I suggest you use ASCII. You don't need decimals, you can encode in hexadecimal with a lookup table.

[andyturk]

https://en.wikipedia.org/wiki/Consistent_Overhead_Byte_Stuffing

[sangar]

Hi Jereon,

Instead sending it in HEX format, do u want me send it in ASCII format ?.If so, it will definitely increase payload size right?.As the Application is timing critical, decoding all the packets back to Hex at PC side and Encoding it in TX side for every transmission will be a big challeng for me.
How about appending Length of payload after HEADER?

[nctnico]

Better add CRC to messages. A simple checksum isn't enough because there are too many ways it can go wrong ('A'+'C' gives the same as 'C'+'A').

[Jeroen3]

It will indeed double in size, 26 bytes payload will become 52 bytes.
In total you will be looking at 52 bytes payload, 2 bytes CRC and 2 bytes flow control. As stunning 56 bytes! At a stunning bitrate of 56 B/s!

At 9600 8N1, which has 960 B/s, it won't be a problem at all.
Encoding/decoding is also way easier than you think.

See my example file of transmitting base16 encoded data. Obviously there can be micro optimizations. Like not using a for loop, and inlining putch.
On the PC side, depending on what environment you are using there might be functions available to encoding/decoding. Try searching for base16.

Output of the example file will be: ?DEADBEEF5D?.

So, unless your PIC is already sweating it's pins out, this should not be very heavy code.

Edit: A library of Base64 you should be able to use on embedded platforms.

[ejeffrey]

I would concur: anything based on proper decoding of a length field is doomed to failure.  One missed or extra byte, and you will be decoding packets incorrectly forever.  It may work in testing, but it is not going to be a reliable protocol.

You need to have some "special" character which the receiver can detect that allows it to synchronize to messages.  Probably the easiest way to do this is to use an ASCII based protocol since you then have all of the standard control codes including NULL, CR/LF, XON/XOFF, and characters > 128 to encode messages start and end points.  It also has the potentially nice benefit that you can interface with a standard terminal program, and also that your host program library (C/C++, python, java, etc) will probably have functions that automatically read a line of text at a time.

If you are worried about computation or communication overhead then you may need a binary protocol, but you should still use something with unambiguous framing like the link that andyturk shared. 

I would say that synchronization is essential in all cases, but error correction depends on your application.  If an erroneous message will cause a problem, then you should have some error detection.  If it is only updating a display that will get overwritten by the next message, you could possibly omit it.  However, a simple CRC or alder checksum is quite easy to implement in a microcontroller.  Even a simple checksum would be helpful.  A checksum can't detect swapped bytes, but honestly that isn't a common error caused by noise -- that is more likely caused by software bugs.

If you need reliable data transmission and you can't ask for a replay, then you need to implement error correction.  Traditional ECC is complicated but low overhead.  Alternately, you can transmit the message 3 times and take the majority.

[mikeselectricstuff]

For a hardwired link you will never see any data errors. CRCs are essential for things like radio links and others where there inherent error sources.

The ONLY thing you need to worry about is that you have correct framing, so you know that if you have the right number of bytes, the message will be correct.
 
There are various ways to ensure framing - the easiest by far is if you can use a specific character ( or sequence of characters) for start of message, which never appears in the payload.

As regards checking, if your code is so tight you can;t do a CRC, do a checksum - the code is trivial - half a dozen assembler instructions or so.
Bear in mind that there may be FIFOs and buffering happenning somewhere, especially with serial over USB, so relying on timing gaps for framing can be problematic, unless the gaps are very long compared to the payload and things like USB forwarding timeouts.

If you are talking to a PC it makes little sense to use low baudrates like 9600 unless you have very long cables.  Using higher baudrates increases the ratio of data to gap, which can make timing based framing more viable. If you know you are talking to a USB interface, you rarely need to stick to "standard"  rates either. rates like 250Kbaud are generally easier on MCUs with standard clock frequencies.
Another advantage of using higher rates is you can use a long unique sequence of characters as a frame-start if you can't impose constraints on the payload.
But as the OP says they are only doing one packet per second, timing based framing would be pretty trivial.

[forrestc]

As I am using Low end Micro-controller (PIC18), I don't want to implement CRC. Suggest if anything which works better.

I want to produce 1-byte CRC for 26 bytes . Can you share sample code?

Responding to the above two items....

First of all a PIC18 will do CRC just fine.   Personally I use CRC16 or CRC32 regularly.   Using a table-based CRC generation code is fast and easy.  8 Bit CRC algorithms are available which are 100% table driven which become really really fast.   Personally, I'd use at least 16 bits of CRC, but that's just me.  Especially if you're sending so little data, you can afford an extra byte for robustness if you really care.   

You should temper the above with what others have said : If this is hardwired and local, the chances of a bit error is minimal at 9600bps, so maybe you really don't need the overhead.  I'm more of a better safe than sorry person myself. 

If you really don't want the overhead of a CRC, but still want some basic checksum, a really simple checksum is to just XOR each byte with the sum of each previous one.   So for each message you start at '0' in your checksum value, and then for each character you do the following:

checksum^=ch;

Instead sending it in HEX format, do u want me send it in ASCII format ?.If so, it will definitely increase payload size right?.As the Application is timing critical, decoding all the packets back to Hex at PC side and Encoding it in TX side for every transmission will be a big challeng for me.
How about appending Length of payload after HEADER?

It is almost always safer to not encode length, but instead some sort of framing indicator.  If you're sending this to a PC, I sure would be tempted to just encode the whole thing in ASCII, and use CRLF as the terminator, so you get strings on the PC which look like:

ff23723af342473f237223a32234ff23723af342473f237223a32234

The advantage of doing this is that it is really easy to separate out messages on the PC -  you're just looking for hexadecimal characters followed by a 0x0d 0x0a.   

Once you've sorted out how to detect a complete message, it is ok to embed frame lengths as part of the message..  But the over the wire protocol should not rely on length to detect the start and and of a frame.

[NorthGuy]

It is nothing safe or reliable in ASCII. Just waste of your time, extra code and extra traffic.

Separators will help you separate your packets. Say,  you dedicate 0xaa to be a separator. You send it before every packet. if you have 0xaa anywhere within a packet, you send it twice.

Decoder receives the stream - if it sees 0xaa, it starts receiving characters until it has 26 of them. If it encounters 0xaa it waits for the next character. If it is 0xaa, it figures this is the second one from doubling 0xaa. If it is not 0xaa then some characters were lost and it assumes that the old packet is screwed and begins a new one.

If something goes haywire, the receiver will synchronize at the beginning of the next packet. In the end you just get packets out of the receiver one by one.

Checking integrity of the packet is the other story. Lots of things you can do. Depends on how noisy is the line and how badly you need the integrity. CRC is good. Doubling the contents of the packet is even better (but takes traffic). Simple checksum might be ok.

[hamster_nz]

If you really care about detecting  errors on the line, rather than ASCII encode 4 bits per byte as follows:

Upper two bits encoded as the upper four bits
0001, 0010, 0100, 1000

Lower two bits encoded as the lower four bits:
1110, 1101, 1011, 0111

(So '0001:1110' is the encoding for '00:00').

Use 00000000 as the start delimiter, and 11111111 as the end delimiter.

You can use 1111:0000 as an idle fill character if you like

That way:

All single bit errors are detected

Quite a few multibit errors are detected

It requires four bit errors to make a valid symbol into the start or end delimiter.

All valid messages have an equal number of ones and zeros.

Better than using ASCII, but way over-thinking it.

Edit: if you interleave bits to can ensure that any burst of two or three errors will result in an invalid symbol, and be detected.

[nctnico]

For a hardwired link you will never see any data errors.

Yes you will! I've built lots of systems which communicate using some kind of protocol over the past decades and they all suffer from malformed packets every once in a while. Just let it run long enough. The key is that the receiver has to be able to re-synchronise itself with the sender. This can be done using special characters (STX, ETX) in an ASCII (text) based protocol for example but error checking is hard to do. Other methods which use a length (2 bytes or more) and CRC16 (or CRC32) also work well. Length is easy to check because the number will need to be less or equal to the maximum packet size.

[matseng]

With a 26 byte datablock being squirted every second there will be a *considerable* period of idle that can be used as a timeout for resynchronizing in case of any lost bytes. There's really no need for any headers, footers or length bytes if it is a fixed length string being sent.

If you really care about the data a suitable CRC is mandatory...  And if you really-really care about the data then add some FEC so you can correct minor bit errors directly and not have to wait for the next data block.
 
This is of course only valid if the baudrate is greater than 300.  At 9600 bps there will be a burst of data lasting 27 milliseconds followed by 973 milliseconds of idle.....

[mikeselectricstuff]

Responding to the above two items...

First of all a PIC18 will do CRC just fine.   Personally I use CRC16 or CRC32 regularly.   Using a table-based CRC generation code is fast and easy.  8 Bit CRC algorithms are available which are 100% table driven which become really really fast.

Sending one small packet a second, it doesn't need to be fast - small code size is probably more important. You can do a serial CRC (or similar) algorithm in maybe 16 instructions while you're waiting for the UART to be ready to send the next character 

[AndyC_772]

For a hardwired link you will never see any data errors. CRCs are essential for things like radio links and others where there inherent error sources.

+1

This is a fact which seems to get overlooked way too often. Within a closed system - say, two adjacent chips on the same PCB - bit errors simply will not occur unless there's something seriously wrong with the design of the board. As long as the protocols used by the two ends of the link are compatible (ie. same data rate and format), and provided there are no silly mistakes in the code, it will work reliably.

You might, if you're designing a highly safety critical system, want to take into account the possibility that errors may occur. Even then, it's more about being able to look someone in the eye and answer "No!" when asked "*Could* your product have been designed to protect against this specific failure method" in a court of law.

If your data is being sent via a link which is known, for good reason, to be noisy, *then* you need to take this into account and work around it... CRCs, retransmission, redundancy, interpolation, that kind of thing. But before you go to all that effort, you should make sure you know exactly what source of noise you're dealing with, what its amplitude is, frequency, repetition rate, burst duration, and so on. That will tell you what the effect is likely to be, and allow you to design a system for error resilience around it.

[hamster_nz]

... 8N1 serial is sometimes ambiguous (unless you have idle time on the link).

Take 01011010110101101011... is it 0101101011 being repeated or 0110101101?

Having something to give confidence in the validity of a message is very useful.

[mikeselectricstuff]

... 8N1 serial is sometimes ambiguous (unless you have idle time on the link).

Take 01011010110101101011... is it 0101101011 being repeated or 0110101101?

Having something to give confidence in the validity of a message is very useful.

Which is why 0xaa is a bad choice of start character. I usually ise 0xff.

[NivagSwerdna]

I have to transmit 26 bytes of payload for every second from PIC18 to PC through Serial communication. Right now , I don't enable parity bit any thing. I frame the payload like below.
As I am using Low end Micro-controller (PIC18), I don't want to implement CRC. Suggest if anything which works better.

Firstly if the distances are short and you don't care about passing taxis and EM pulses from nuclear explosions then don't worry about it.  (Use shielded cables where you can)
Secondly, there are two levels to consider... you can add parity at the character level or you can add an application CRC.
For parity probably not worth it.. stick with 8-N-1
For the application level... you can not bother (and take the risk) or instead add something simple... you can add the bytes (overflowing as you go) and send the result... you can XOR the bytes... or you can use a traditional CRC (e.g. http://ww1.microchip.com/downloads/en/AppNotes/01148a.pdf).  Although CRCs look computationally complex they are actually cheap since they are only binary operations, shifts etc, easily do-able on the lowest processor.
The final gotcha is the clock stability... the sender and receiver clocks will differ (and may drift due to temperature etc) especially if using a resonator rather a crystal... so keep your speeds within your design constraints... probably not a real problem for you.
Have fun!

PS
For the data itself you need a frame to allow you to detect the start... i.e.

<SYNC> data.... <CRC>

Quite a nice format is...

<SYNC> <VERSION> <OPERATION> <LENGTH> .... data... <CRC>

where say SYNC is 01 (SOH) (and value 01 and SYNC are disambiguated using byte stuffing)

<VERSION> e.g. 01 can be handy if you have many devices and the protocol changes over time (saved me from trouble before!)
<OPERATION> is say 01 Data Reading from Brain Probe

Historically the STX, SOH, ETX etc characters were used or exactly these framing purposes.... "back in the day" 

[NorthGuy]

Which is why 0xaa is a bad choice of start character. I usually ise 0xff.

If you don't have gaps between packets, 0xff is certainly a better choice for UART.

[nctnico]

If your data is being sent via a link which is known, for good reason, to be noisy, *then* you need to take this into account and work around it... CRCs, retransmission, redundancy, interpolation, that kind of thing. But before you go to all that effort, you should make sure you know exactly what source of noise you're dealing with, what its amplitude is, frequency, repetition rate, burst duration, and so on. That will tell you what the effect is likely to be, and allow you to design a system for error resilience around it.

The problem is that you don't know that in advance. At some point I had to change a circuit because it started misbehaving when it was mounted near a series motor with worn brushes. This device had passed EMC testing already! IMHO creating a communication protocol without contents checking (CRC), re-synchronisation and re-transmit facilities is asking for unreliable operation.

[Kjelt]

I also add an <end of packet> byte for robustness and a timegap between two packets >2 bytes Timing.
So if the lenght exceeds the lenght byte or the time that a byte was received exceeds one and half byte, i drop the packet and wait for a new header. Since I use acknowledgments on each transfer the master knows if the transmission failed.

[Delta]

With only 26 bytes to send each second, just keep blootering the same packet over the link. If your receiver gets 1 packet differing from the other 23, it isn't rocket surgery to work out what is good data and what is noise.

[Pack34]

Will you be sending multiple types of packets at different times? Or will you be streaming a simple and standard control message? You could simply send

START BYTE (0xFF)
VAL1
VAL2
VAL3
...
VALN
CHECKSUM
END BYTE (0xFE)

Then just be sure all of the payload and checkum bytes are properly masked such that your START and END bytes cannot appear. This way you simply spool until you see 0xFF appear on the bus, read until you see 0xFE, validate the checksum, then parse your internal bytes.

Quick and dirty.

[Kjelt]

That makes no sense, when transferring data all 255 values could be valid. Just add a length field(byte) after the start byte and the protocol is robust enough.
Already use something similar for over twenty years 24/7 works like a charm, at least if you can write code.

[Kalvin]

I typically prefer to send the length field as non-inverted and inverted. That will provide a fast way to detect whether the packet length is valid or not. There may be some better ways to improve the error detection of the length field by using some clever coding, something similar to what hamster_nz suggested in his earlier post https://www.eevblog.com/forum/microcontrollers/how-should-be-a-serial-communication-packet-format/msg1257528/#msg1257528.

[Hensingler]

That makes no sense, when transferring data all 255 values could be valid. Just add a length field(byte) after the start byte and the protocol is robust enough.

Any protocol depending on a length field is a fail. You don't know you have a valid length field without checking the CRC and you can't check the CRC without a valid length field. A length field corrupted long means you might read in the next several packets before checking something that isn't a CRC.

So you then have to start adding time outs and the associated real time requirements for the transmitter (like you already posted you did) because you chose dumb protocol to start with.

[nctnico]

That makes no sense, when transferring data all 255 values could be valid. Just add a length field(byte) after the start byte and the protocol is robust enough.

Any protocol depending on a length field is a fail. You don't know you have a valid length field without checking the CRC and you can't check the CRC without a valid length field. A length field corrupted long means you might read in the next several packets before checking something that isn't a CRC.

So you then have to start adding time outs and the associated real time requirements for the transmitter (like you already posted you did) because you chose dumb protocol to start with.

That is only true if the transmitter sends packets before it receives a reply (at least an acknowledge). So far the only protocol I have come across which does that is TCP/IP but that sits on top of a link layer which already does length and CRC checking.

[Kjelt]

you chose a dumb protocol

Pffff i like the KISS principle, if it works, it works. If it fails even after a year I analyze and change. If youwant robustness to the max
then create a header with its own checksum if you are so paranoid like the ip protocol and be done with it. My system is wired rs485 and have not had a single fail in 20 years with my dumb protocol. Design your own and be happy, i am happy with my choice.

[nfmax]

That makes no sense, when transferring data all 255 values could be valid. Just add a length field(byte) after the start byte and the protocol is robust enough.

Any protocol depending on a length field is a fail. You don't know you have a valid length field without checking the CRC and you can't check the CRC without a valid length field. A length field corrupted long means you might read in the next several packets before checking something that isn't a CRC.

Real communications protocols get round this using a fixed-length header containing the packet size field, with a header checksum.

The OP is better off using an already existing protocol, rather than trying to invent one just for this purpose, even if the existing protocol is massive overkill for the job. In another galaxy, long, long ago, we did just this. Used the Kermit protocol between a hand-held controller and a data acquisition system. This had to operate in the carrier flight deck environment, 200V/m EMF, so we needed good error correction. Nobody needed to know what the protocol was (we made both boxes) so we just kept schtumm about it!

[Hensingler]

That is only true if the transmitter sends packets before it receives a reply (at least an acknowledge).

If the transmitter doesn't send more packets the receiver will wait for ever to receive the corrupted long length making the time outs mandatory. There no requirement for replies anyway. I have implemented several unidirectional serial links and networks where only commands which queried information produced response messages, in particular broadcast messages didn't.

Real communications protocols get round this using a fixed-length header containing the packet size field, with a header checksum.

Which is just pointless overhead and less reliable than using unique message separator or start/stop symbols. If you get a break in the comms link (quite common for radio) you will still be sitting there waiting for your validated length message that will never come (and make time outs a requirement).

 

[ejeffrey]

you chose a dumb protocol

Pffff i like the KISS principle, if it works, it works. If it fails even after a year I analyze and change. If youwant robustness to the max
then create a header with its own checksum if you are so paranoid like the ip protocol and be done with it. My system is wired rs485 and have not had a single fail in 20 years with my dumb protocol. Design your own and be happy, i am happy with my choice.

The simple thing is to use one of the existing protocols that already solves this, not rolling your own either way.  And honestly, I don't believe that you have never had a problem if you really did what you describe.

Serial ports can and will generate garbage bytes when the microcontroller resets.  The user will restart the host application in the middle of a message.  If you have a serial cable with a connector, someone will disconnect and reconnect the port.  The problem with using length bytes for framing is that if you loose your frame pointer, you are lost forever until you restart the whole system.  So I don't believe that you have used this protocol with out any way to recover synchronization.  Your way is probably just not as good as a protocol that auto synchronizes every message.  There is no need to use a protocol that doesn't support this.  It is trivial to dedicate a character to frame boundaries and then escape it when it occurs in data.  If you really care about the worst case overhead where every byte in a message has to be escaped, and you can't just increase your line rate, then use a slightly more complicated protocol.

All more modern / higher level communications interfaces have built in ways to do this.  Even GPIB, which is nearly as ancient as RS232 has this with the EOI line.  Modern protocols like ethernet, USB, etc. all have built-in message framing.  Hell, even async serial actually has this capability, you can generate a BREAK to synchronize the sender and receiver, except that handling of BREAK condition is pretty hit or miss between implementations.

[Kjelt]

Too long to type here on an ipad in hotel but no i have no problems since i do have synchronization start stop frame bytes length, command and crc. I have 30 slaves and one master , can do individual control as well as broadcast and it works, if it did not i would notice since they are the lights in my livingroom and other things as blends for screens. The reason i definitely wanted a length field is to accomodate new additions that might require a longer or shorter messages and i wanted future expansion  flexibility without having to reflash all firmwares from other slaves.

I agree that whoever wants to do this should re-use existing protocols, however for me it was part of the fun to design my own and basically i found many other similar protocols from others.
If you only copy that which others have made then you will not learn new things.
One of the protocols i came across was a fixed length protocol. So frames were always the same size either too long or too short and inflexible as you can think of. If that fits your design go do it. If you want to reserve some of the bytevalues for your protocol do it, i don't want that, if I sniff my protocol i can read it , i don't have to decode or recalculate it. If i want to encrypt my payload i can do it, i don't have to worry that some bytevalues are reserved.

[mikeselectricstuff]

  Hell, even async serial actually has this capability, you can generate a BREAK to synchronize the sender and receiver, except that handling of BREAK condition is pretty hit or miss between implementations.

And can get even more interesting when the receive UART has a FIFO. I don't recall ever seeing an MCU datasheet that clearly explained exactly how its FIFO'd UART deals with break conditions.
The other problem with out-of-band features like breaks is it gets messy when somewhere down the line you need to transport the UART stream over another protocol like USB or ethernet.

[mikerj]

Which is just pointless overhead and less reliable than using unique message separator or start/stop symbols. If you get a break in the comms link (quite common for radio) you will still be sitting there waiting for your validated length message that will never come (and make time outs a requirement).

If your unique message separator gets corrupted, what stops the receiver waiting for the end of of a packet which it's already missed, if timeouts are not an option?

[Naguissa]

Header - length - base64(data) - crc (also base64) - end mark.

Header and end should be non-base64 symbols.

Edit: one note, length is for 2nd level error checking, not for framing.

Enviado desde mi Jolla mediante Tapatalk

[Hensingler]

If your unique message separator gets corrupted, what stops the receiver waiting for the end of of a packet which it's already missed, if timeouts are not an option?

It is waiting, it didn't receive a corrupt packet so what. At all times receiving a start symbol restarts collection of a packet. The start/stop or separator scheme guarantees that a received sequence of bytes forming a valid packet will *always* be interpreted as a valid packet regardless previous state. Something which protocols with length fields and time outs do not.

As an aside you can not make any assumption about a sequence of bytes which is not a valid checksum verified packet - it can not be relied on to have more meaning than received noise or received nothing. If you need to look at it any of it before verification your protocol is broken.

Header - length - base64(data) - crc (also base64) - end mark.

Settle for a constant 25% base64 encoding overhead while a 3 'special' symbol start/stop/quote scheme gives a variable 1.2% average overhead and much simpler implementation.

[krho]

Settle for a constant 25% base64 encoding overhead while a 3 'special' symbol start/stop/quote scheme gives a variable 1.2% average overhead and much simpler implementation.

 

Or even better

[Jeroen3]

Put it in 9 bits mode. Enable 9th bit on control characters.

[Karel]

And the winner is.....

https://www.rfc-editor.org/rfc/rfc1055.txt

https://en.wikipedia.org/wiki/Serial_Line_Internet_Protocol

Simple, efficient and elegant, including sourcecode.

[voltsandjolts]

And the winner is.....

https://en.wikipedia.org/wiki/Consistent_Overhead_Byte_Stuffing

http://stuartcheshire.org/papers/COBSforToN.pdf

Simple, very efficient and elegant, including sourcecode.

[Jeroen3]

Maybe we should make a better one?

[veryevil]

I'd like to put in another vote for SLIP. used it many times in the past. Easy to encode. Easy to decode. Any length you want.

Code: [Select]

#define SLIP_END 0xC0
#define SLIP_ESC 0xDB
#define ESC_END 0xDC
#define ESC_ESC 0xDD


void EncodeSlip(char *buffer_ptr, unsigned int buffer_length_bytes)
{
unsigned int i = 0;
printf("%c", SLIP_END);            // Write start of SLIP frame   

while (i < (buffer_length_bytes)) {          // Send all buffer in SLIP format   
switch (buffer_ptr[i]) {                   // check to see if is a special character   
case SLIP_END:   
printf("%c%c", SLIP_ESC, ESC_END);     // escape special character   
break;   

case SLIP_ESC:   
printf("%c%c", SLIP_ESC, ESC_ESC);     // escape special character   
break;   

default:   
printf("%c", buffer_ptr[i]);        // send raw character   
}   
i++;   // continue with next character send   
}   
printf ("%c", SLIP_END);
}



[voltsandjolts]

SLIP is nice and simple. However, it can create a lot of overhead (potentially double the number of bytes to transfer) if you are sending a lot of bytes which happen to be the same as SLIP_END. If you need to maintain transfer rates with minimal overhead regardless of data content, then use COBS (links in my post above).

[Karel]

When using SLIP, it can be convenient to change the SLIP_END character to 0x0A (newline).
This is handy for testing and sending commands manually via a terminal (no need to encode the commands).

[andyturk]

It is trivial to dedicate a character to frame boundaries and then escape it when it occurs in data.  If you really care about the worst case overhead where every byte in a message has to be escaped, and you can't just increase your line rate, then use a slightly more complicated protocol.

Bingo.

COBS allows you to unambiguously dedicate a single octet (byte) to framing. The sender encodes every message (whatever the length) using COBS and appends the framing octet (e.g., 0x00) to the end. The receiver reads everything up to a 0x00 into a buffer and reverses the COBS encoding to recover the original message.

Essentially what COBS does is guarantee that 0x00 *never* occurs in encoded form (with only space 0.4% overhead). That's how you can use 0x00 on the receiving side to delimit your frames. COBS deals with frames of arbitrary length and gives you the option of dispensing with an explicit length within the frame itself.

More info here: https://www.embeddedrelated.com/showarticle/113.php

EDIT: I see voltsandjolts is pounding the same drum. 

[Karel]

What I don't like about COBS:

- max packet size is limited to 254 bytes

- needs to buffer the whole packet at the transmitter end. Not nice when using a small mcu

[voltsandjolts]

Incorrect -  max packet length is not limited to 254 bytes. Its arbitrary length.

Yes, a micro needs a 256 byte (plus a few bytes extra) buffer to send COBS but that is no problem in these days of cheap ARM Cortex.

Edit: Smaller buffer can be used depending on the max packet length you want to send i.e. packet length + overhead size (0.5% or so) for less than 256 byte packet.

[Karel]

Incorrect -  max packet length is not limited to 254 bytes. Its arbitrary length.

"COBS transforms a data set of up to 254 bytes..."

"In the case of byte streams, or fixed-size data sets larger than 254 bytes,
COBS requires data to be encoded a section at a time, such that no section exceeds 254 bytes in size."

https://en.wikipedia.org/wiki/Consistent_Overhead_Byte_Stuffing#Packet_framing_and_stuffing


"One aspect of COBS that differs from conventional two-for-one substitution encodings like PPP is that
PPP operates on a single byte at a time, whereas COBS has to 'look ahead' up to 254 bytes.
In the context of thinking about network data as a continuous stream of bytes,
this suggests that COBS requires more buffering and consequently adds more delay to the stream than PPP does"

http://www.stuartcheshire.org/papers/COBSforSIGCOMM/

[voltsandjolts]

Take whatever data length you have and send it by processing it up to 254 bytes at a time. That's just the max length until you must insert a COBS code byte of 0xFF. Then immediately start the next chunk, you don't have to terminate the packet. Read the source code in the Cheshire & Baker paper I linked.

[smoothVTer]

As I am using Low end Micro-controller (PIC18), I don't want to implement CRC. Suggest if anything which works better.

I want to produce 1-byte CRC for 26 bytes . Can you share sample code?

Responding to the above two items....

First of all a PIC18 will do CRC just fine.   Personally I use CRC16 or CRC32 regularly.   Using a table-based CRC generation code is fast and easy.  8 Bit CRC algorithms are available which are 100% table driven which become really really fast.   Personally, I'd use at least 16 bits of CRC, but that's just me.  Especially if you're sending so little data, you can afford an extra byte for robustness if you really care.   

You should temper the above with what others have said : If this is hardwired and local, the chances of a bit error is minimal at 9600bps, so maybe you really don't need the overhead.  I'm more of a better safe than sorry person myself. 

If you really don't want the overhead of a CRC, but still want some basic checksum, a really simple checksum is to just XOR each byte with the sum of each previous one.   So for each message you start at '0' in your checksum value, and then for each character you do the following:

checksum^=ch;

Instead sending it in HEX format, do u want me send it in ASCII format ?.If so, it will definitely increase payload size right?.As the Application is timing critical, decoding all the packets back to Hex at PC side and Encoding it in TX side for every transmission will be a big challeng for me.
How about appending Length of payload after HEADER?

It is almost always safer to not encode length, but instead some sort of framing indicator.  If you're sending this to a PC, I sure would be tempted to just encode the whole thing in ASCII, and use CRLF as the terminator, so you get strings on the PC which look like:

ff23723af342473f237223a32234ff23723af342473f237223a32234

The advantage of doing this is that it is really easy to separate out messages on the PC -  you're just looking for hexadecimal characters followed by a 0x0d 0x0a.   

Once you've sorted out how to detect a complete message, it is ok to embed frame lengths as part of the message..  But the over the wire protocol should not rely on length to detect the start and and of a frame.

As an aside ... have you been able to get the hardware-based PIC18 CRC to work?   I've struggled with it for several days on and off, and never got it to work;  The register that stores the CRC is always wrong in my case.   I followed every word of the user guide/data sheet on the hardware CRC, checked the eratta ... no luck.  I ended up going with the table-based CRC + algorithm and it worked correctly the first time.     Is there a general trick to using the hardware CRC?

[magetoo]

What kind of "correctness" do you need?  What is the worst case scenario if a faulty packet is received?  If the worst that can happen is something like a heater turning on when it shouldn't for one second, wasting a miniscule amount of electricity, then don't bother with anything fancy.  The next packet received will fix it.

But if you really need near-perfect reliability, then do look into error-correcting codes.  I would suggest the book A Commonsense Approach to the Theory of Error Correcting Codes by Benjamin Arazi (MIT Press, 1988; ISBN 0-262-01098-4), it's relatively free of higher math and shows how to build CRC generators, Hamming and Reed-Solomon encoders and decoders using just shift registers and XOR gates.  (I haven't written any code, but I don't see how a few bit shifts and XOR operations can be that taxing for even a low-end micro.)

Turns out a lot of this stuff can be fairly simple in practice, it's just the explanations and jargon that is overly complicated.

[cellularmitosis]

I just recently implemented a packet format with CRC8 between an Arduino and a Python script on the PC.  Thought I'd share a specific worked example:

https://github.com/pepaslabs/lab-logger/blob/78ae579e09cad7eea7615533d5952381e60a404a/client-script/Si7021-logger.py

https://github.com/pepaslabs/lab-logger/tree/78ae579e09cad7eea7615533d5952381e60a404a/firmware

[sangar]

Hi Jeroen3,
Can you explain me that how did you calculate CRC7_POLY = 0x91 in the below clode.

Code: [Select]

unsigned char crc8(unsigned char message[], unsigned char length)
{
    const unsigned char CRC7_POLY = 0x91;
    unsigned char i, j, crc = 0;
    for (i = 0; i < length; i++){
        crc ^= message[i];
        for (j = 0; j < 8; j++)
        {
            if (crc & 1)
                crc ^= CRC7_POLY;
            crc >>= 1;
        }
    }
    return crc;
}


Regards,
Muthu

[Jeroen3]

Some code you write steal once.

[TomS_]

"COBS transforms a data set of up to 254 bytes..."

254 contiguous non-zero bytes... specifically.

With the right byte pattern, the length can be arbitrary as stated. All you need is a zero byte within every 254 bytes to reset the pointer to the next "interpreted byte" and you can go on and on for as long as you like.

Ive only just discovered COBS through this thread, and it looks really neat. Im now considering implementing it in to a project I am working on.

edit: or as voltsandjolts says, the 0xFF "special case" (still learning about it!)

[voltsandjolts]

With the right byte pattern, the length can be arbitrary as stated. All you need is a zero byte within every 254 bytes to reset the pointer to the next "interpreted byte" and you can go on and on for as long as you like.

No, you don't need a zero byte. The input data can be any stream of bytes.

[TomS_]

No, you don't need a zero byte. The input data can be any stream of bytes.

Yeah that was based on an initial understanding, but as I read more and more I understood more and more.

I left everything in with an edit at the end because Im not trying to hide that.