How should be a serial communication packet format ?

[sangar]

Hello all,

I have to transmit 26 bytes of payload for every second from PIC18 to PC through Serial communication. Right now , I don't enable parity bit any thing. I frame the payload like below.


Header(1-byte)     Length(1-byte)   Payload      Footer(2-byte).


Is the above safe?


As I am using Low end Micro-controller (PIC18), I don't want to implement CRC. Suggest if anything which works better.



Thanks,
Sangar 

[krho]

If you don't want to implement the CRC because of space constraint then you can implement fletcher16. This one is simple and small

[sangar]

Hi,
Thanks.


Fletcher-16  will give 2 bytes of CRC ?

I want to produce 1-byte CRC for 26 bytes . Can you share sample code?

[Rerouter]

It comes down to if every packet being accurate matters to you, if you want to catch a certain amount of errors, you can use a parity bit or an xor check,

[voltsandjolts]

This might help you choose:
https://users.ece.cmu.edu/~koopman/pubs/KoopmanCRCWebinar9May2012.pdf

[cv007]

or just send it twice- if 2 'packets' are not identical, then there is a problem. 26 extra bytes does not take a lot of time to send and you already have the code to send it once, so just send it twice.  208 bits of parity

[Jeroen3]

How will you ensure synchronisation? What if the header/footer appear in the payload?
With this protocol, you will have use some timeout mechanism to indicate a new packet is arriving.
Eg, what if something goes wrong?:
header - payload - footer ...... header - pa.. [interruption] hdr - payload - footer ...

I have made a reader for a device that used this kind of protocol. Packets would start with 0xAA, some payload of variable length, and end with a CRC.
It was impossible to keep it synchronized, the pc won't give nice full packets in the received events.
I ended up to try to decode a packet and check crc each 0xAA that appeared in the stream. An awfully inefficient method.

When working with streams, I prefer to use ascii since you can use dedicated flow control characters. It makes the implementation easier and more robust, at the cost of troughput.
Since you're going to the PC, at a very low rate, I suggest you use ASCII. You don't need decimals, you can encode in hexadecimal with a lookup table.

[andyturk]

https://en.wikipedia.org/wiki/Consistent_Overhead_Byte_Stuffing

[sangar]

Hi Jereon,

Instead sending it in HEX format, do u want me send it in ASCII format ?.If so, it will definitely increase payload size right?.As the Application is timing critical, decoding all the packets back to Hex at PC side and Encoding it in TX side for every transmission will be a big challeng for me.
How about appending Length of payload after HEADER?

[nctnico]

Better add CRC to messages. A simple checksum isn't enough because there are too many ways it can go wrong ('A'+'C' gives the same as 'C'+'A').

[Jeroen3]

It will indeed double in size, 26 bytes payload will become 52 bytes.
In total you will be looking at 52 bytes payload, 2 bytes CRC and 2 bytes flow control. As stunning 56 bytes! At a stunning bitrate of 56 B/s!

At 9600 8N1, which has 960 B/s, it won't be a problem at all.
Encoding/decoding is also way easier than you think.

See my example file of transmitting base16 encoded data. Obviously there can be micro optimizations. Like not using a for loop, and inlining putch.
On the PC side, depending on what environment you are using there might be functions available to encoding/decoding. Try searching for base16.

Output of the example file will be: ?DEADBEEF5D?.

So, unless your PIC is already sweating it's pins out, this should not be very heavy code.

Edit: A library of Base64 you should be able to use on embedded platforms.

[ejeffrey]

I would concur: anything based on proper decoding of a length field is doomed to failure.  One missed or extra byte, and you will be decoding packets incorrectly forever.  It may work in testing, but it is not going to be a reliable protocol.

You need to have some "special" character which the receiver can detect that allows it to synchronize to messages.  Probably the easiest way to do this is to use an ASCII based protocol since you then have all of the standard control codes including NULL, CR/LF, XON/XOFF, and characters > 128 to encode messages start and end points.  It also has the potentially nice benefit that you can interface with a standard terminal program, and also that your host program library (C/C++, python, java, etc) will probably have functions that automatically read a line of text at a time.

If you are worried about computation or communication overhead then you may need a binary protocol, but you should still use something with unambiguous framing like the link that andyturk shared. 

I would say that synchronization is essential in all cases, but error correction depends on your application.  If an erroneous message will cause a problem, then you should have some error detection.  If it is only updating a display that will get overwritten by the next message, you could possibly omit it.  However, a simple CRC or alder checksum is quite easy to implement in a microcontroller.  Even a simple checksum would be helpful.  A checksum can't detect swapped bytes, but honestly that isn't a common error caused by noise -- that is more likely caused by software bugs.

If you need reliable data transmission and you can't ask for a replay, then you need to implement error correction.  Traditional ECC is complicated but low overhead.  Alternately, you can transmit the message 3 times and take the majority.

[mikeselectricstuff]

For a hardwired link you will never see any data errors. CRCs are essential for things like radio links and others where there inherent error sources.

The ONLY thing you need to worry about is that you have correct framing, so you know that if you have the right number of bytes, the message will be correct.
 
There are various ways to ensure framing - the easiest by far is if you can use a specific character ( or sequence of characters) for start of message, which never appears in the payload.

As regards checking, if your code is so tight you can;t do a CRC, do a checksum - the code is trivial - half a dozen assembler instructions or so.
Bear in mind that there may be FIFOs and buffering happenning somewhere, especially with serial over USB, so relying on timing gaps for framing can be problematic, unless the gaps are very long compared to the payload and things like USB forwarding timeouts.

If you are talking to a PC it makes little sense to use low baudrates like 9600 unless you have very long cables.  Using higher baudrates increases the ratio of data to gap, which can make timing based framing more viable. If you know you are talking to a USB interface, you rarely need to stick to "standard"  rates either. rates like 250Kbaud are generally easier on MCUs with standard clock frequencies.
Another advantage of using higher rates is you can use a long unique sequence of characters as a frame-start if you can't impose constraints on the payload.
But as the OP says they are only doing one packet per second, timing based framing would be pretty trivial.

[forrestc]

As I am using Low end Micro-controller (PIC18), I don't want to implement CRC. Suggest if anything which works better.

I want to produce 1-byte CRC for 26 bytes . Can you share sample code?

Responding to the above two items....

First of all a PIC18 will do CRC just fine.   Personally I use CRC16 or CRC32 regularly.   Using a table-based CRC generation code is fast and easy.  8 Bit CRC algorithms are available which are 100% table driven which become really really fast.   Personally, I'd use at least 16 bits of CRC, but that's just me.  Especially if you're sending so little data, you can afford an extra byte for robustness if you really care.   

You should temper the above with what others have said : If this is hardwired and local, the chances of a bit error is minimal at 9600bps, so maybe you really don't need the overhead.  I'm more of a better safe than sorry person myself. 

If you really don't want the overhead of a CRC, but still want some basic checksum, a really simple checksum is to just XOR each byte with the sum of each previous one.   So for each message you start at '0' in your checksum value, and then for each character you do the following:

checksum^=ch;

Instead sending it in HEX format, do u want me send it in ASCII format ?.If so, it will definitely increase payload size right?.As the Application is timing critical, decoding all the packets back to Hex at PC side and Encoding it in TX side for every transmission will be a big challeng for me.
How about appending Length of payload after HEADER?

It is almost always safer to not encode length, but instead some sort of framing indicator.  If you're sending this to a PC, I sure would be tempted to just encode the whole thing in ASCII, and use CRLF as the terminator, so you get strings on the PC which look like:

ff23723af342473f237223a32234ff23723af342473f237223a32234

The advantage of doing this is that it is really easy to separate out messages on the PC -  you're just looking for hexadecimal characters followed by a 0x0d 0x0a.   

Once you've sorted out how to detect a complete message, it is ok to embed frame lengths as part of the message..  But the over the wire protocol should not rely on length to detect the start and and of a frame.

[NorthGuy]

It is nothing safe or reliable in ASCII. Just waste of your time, extra code and extra traffic.

Separators will help you separate your packets. Say,  you dedicate 0xaa to be a separator. You send it before every packet. if you have 0xaa anywhere within a packet, you send it twice.

Decoder receives the stream - if it sees 0xaa, it starts receiving characters until it has 26 of them. If it encounters 0xaa it waits for the next character. If it is 0xaa, it figures this is the second one from doubling 0xaa. If it is not 0xaa then some characters were lost and it assumes that the old packet is screwed and begins a new one.

If something goes haywire, the receiver will synchronize at the beginning of the next packet. In the end you just get packets out of the receiver one by one.

Checking integrity of the packet is the other story. Lots of things you can do. Depends on how noisy is the line and how badly you need the integrity. CRC is good. Doubling the contents of the packet is even better (but takes traffic). Simple checksum might be ok.

[hamster_nz]

If you really care about detecting  errors on the line, rather than ASCII encode 4 bits per byte as follows:

Upper two bits encoded as the upper four bits
0001, 0010, 0100, 1000

Lower two bits encoded as the lower four bits:
1110, 1101, 1011, 0111

(So '0001:1110' is the encoding for '00:00').

Use 00000000 as the start delimiter, and 11111111 as the end delimiter.

You can use 1111:0000 as an idle fill character if you like

That way:

All single bit errors are detected

Quite a few multibit errors are detected

It requires four bit errors to make a valid symbol into the start or end delimiter.

All valid messages have an equal number of ones and zeros.

Better than using ASCII, but way over-thinking it.

Edit: if you interleave bits to can ensure that any burst of two or three errors will result in an invalid symbol, and be detected.

[nctnico]

For a hardwired link you will never see any data errors.

Yes you will! I've built lots of systems which communicate using some kind of protocol over the past decades and they all suffer from malformed packets every once in a while. Just let it run long enough. The key is that the receiver has to be able to re-synchronise itself with the sender. This can be done using special characters (STX, ETX) in an ASCII (text) based protocol for example but error checking is hard to do. Other methods which use a length (2 bytes or more) and CRC16 (or CRC32) also work well. Length is easy to check because the number will need to be less or equal to the maximum packet size.

[matseng]

With a 26 byte datablock being squirted every second there will be a *considerable* period of idle that can be used as a timeout for resynchronizing in case of any lost bytes. There's really no need for any headers, footers or length bytes if it is a fixed length string being sent.

If you really care about the data a suitable CRC is mandatory...  And if you really-really care about the data then add some FEC so you can correct minor bit errors directly and not have to wait for the next data block.
 
This is of course only valid if the baudrate is greater than 300.  At 9600 bps there will be a burst of data lasting 27 milliseconds followed by 973 milliseconds of idle.....

[mikeselectricstuff]

Responding to the above two items...

First of all a PIC18 will do CRC just fine.   Personally I use CRC16 or CRC32 regularly.   Using a table-based CRC generation code is fast and easy.  8 Bit CRC algorithms are available which are 100% table driven which become really really fast.

Sending one small packet a second, it doesn't need to be fast - small code size is probably more important. You can do a serial CRC (or similar) algorithm in maybe 16 instructions while you're waiting for the UART to be ready to send the next character 

[AndyC_772]

For a hardwired link you will never see any data errors. CRCs are essential for things like radio links and others where there inherent error sources.

+1

This is a fact which seems to get overlooked way too often. Within a closed system - say, two adjacent chips on the same PCB - bit errors simply will not occur unless there's something seriously wrong with the design of the board. As long as the protocols used by the two ends of the link are compatible (ie. same data rate and format), and provided there are no silly mistakes in the code, it will work reliably.

You might, if you're designing a highly safety critical system, want to take into account the possibility that errors may occur. Even then, it's more about being able to look someone in the eye and answer "No!" when asked "*Could* your product have been designed to protect against this specific failure method" in a court of law.

If your data is being sent via a link which is known, for good reason, to be noisy, *then* you need to take this into account and work around it... CRCs, retransmission, redundancy, interpolation, that kind of thing. But before you go to all that effort, you should make sure you know exactly what source of noise you're dealing with, what its amplitude is, frequency, repetition rate, burst duration, and so on. That will tell you what the effect is likely to be, and allow you to design a system for error resilience around it.

[hamster_nz]

... 8N1 serial is sometimes ambiguous (unless you have idle time on the link).

Take 01011010110101101011... is it 0101101011 being repeated or 0110101101?

Having something to give confidence in the validity of a message is very useful.

[mikeselectricstuff]

... 8N1 serial is sometimes ambiguous (unless you have idle time on the link).

Take 01011010110101101011... is it 0101101011 being repeated or 0110101101?

Having something to give confidence in the validity of a message is very useful.

Which is why 0xaa is a bad choice of start character. I usually ise 0xff.

[NivagSwerdna]

I have to transmit 26 bytes of payload for every second from PIC18 to PC through Serial communication. Right now , I don't enable parity bit any thing. I frame the payload like below.
As I am using Low end Micro-controller (PIC18), I don't want to implement CRC. Suggest if anything which works better.

Firstly if the distances are short and you don't care about passing taxis and EM pulses from nuclear explosions then don't worry about it.  (Use shielded cables where you can)
Secondly, there are two levels to consider... you can add parity at the character level or you can add an application CRC.
For parity probably not worth it.. stick with 8-N-1
For the application level... you can not bother (and take the risk) or instead add something simple... you can add the bytes (overflowing as you go) and send the result... you can XOR the bytes... or you can use a traditional CRC (e.g. http://ww1.microchip.com/downloads/en/AppNotes/01148a.pdf).  Although CRCs look computationally complex they are actually cheap since they are only binary operations, shifts etc, easily do-able on the lowest processor.
The final gotcha is the clock stability... the sender and receiver clocks will differ (and may drift due to temperature etc) especially if using a resonator rather a crystal... so keep your speeds within your design constraints... probably not a real problem for you.
Have fun!

PS
For the data itself you need a frame to allow you to detect the start... i.e.

<SYNC> data.... <CRC>

Quite a nice format is...

<SYNC> <VERSION> <OPERATION> <LENGTH> .... data... <CRC>

where say SYNC is 01 (SOH) (and value 01 and SYNC are disambiguated using byte stuffing)

<VERSION> e.g. 01 can be handy if you have many devices and the protocol changes over time (saved me from trouble before!)
<OPERATION> is say 01 Data Reading from Brain Probe

Historically the STX, SOH, ETX etc characters were used or exactly these framing purposes.... "back in the day" 

[NorthGuy]

Which is why 0xaa is a bad choice of start character. I usually ise 0xff.

If you don't have gaps between packets, 0xff is certainly a better choice for UART.

[nctnico]

If your data is being sent via a link which is known, for good reason, to be noisy, *then* you need to take this into account and work around it... CRCs, retransmission, redundancy, interpolation, that kind of thing. But before you go to all that effort, you should make sure you know exactly what source of noise you're dealing with, what its amplitude is, frequency, repetition rate, burst duration, and so on. That will tell you what the effect is likely to be, and allow you to design a system for error resilience around it.

The problem is that you don't know that in advance. At some point I had to change a circuit because it started misbehaving when it was mounted near a series motor with worn brushes. This device had passed EMC testing already! IMHO creating a communication protocol without contents checking (CRC), re-synchronisation and re-transmit facilities is asking for unreliable operation.