How should be a serial communication packet format ?

[Karel]

What I don't like about COBS:

- max packet size is limited to 254 bytes

- needs to buffer the whole packet at the transmitter end. Not nice when using a small mcu

[voltsandjolts]

Incorrect -  max packet length is not limited to 254 bytes. Its arbitrary length.

Yes, a micro needs a 256 byte (plus a few bytes extra) buffer to send COBS but that is no problem in these days of cheap ARM Cortex.

Edit: Smaller buffer can be used depending on the max packet length you want to send i.e. packet length + overhead size (0.5% or so) for less than 256 byte packet.

[Karel]

Incorrect -  max packet length is not limited to 254 bytes. Its arbitrary length.

"COBS transforms a data set of up to 254 bytes..."

"In the case of byte streams, or fixed-size data sets larger than 254 bytes,
COBS requires data to be encoded a section at a time, such that no section exceeds 254 bytes in size."

https://en.wikipedia.org/wiki/Consistent_Overhead_Byte_Stuffing#Packet_framing_and_stuffing


"One aspect of COBS that differs from conventional two-for-one substitution encodings like PPP is that
PPP operates on a single byte at a time, whereas COBS has to 'look ahead' up to 254 bytes.
In the context of thinking about network data as a continuous stream of bytes,
this suggests that COBS requires more buffering and consequently adds more delay to the stream than PPP does"

http://www.stuartcheshire.org/papers/COBSforSIGCOMM/

[voltsandjolts]

Take whatever data length you have and send it by processing it up to 254 bytes at a time. That's just the max length until you must insert a COBS code byte of 0xFF. Then immediately start the next chunk, you don't have to terminate the packet. Read the source code in the Cheshire & Baker paper I linked.

[smoothVTer]

As I am using Low end Micro-controller (PIC18), I don't want to implement CRC. Suggest if anything which works better.

I want to produce 1-byte CRC for 26 bytes . Can you share sample code?

Responding to the above two items....

First of all a PIC18 will do CRC just fine.   Personally I use CRC16 or CRC32 regularly.   Using a table-based CRC generation code is fast and easy.  8 Bit CRC algorithms are available which are 100% table driven which become really really fast.   Personally, I'd use at least 16 bits of CRC, but that's just me.  Especially if you're sending so little data, you can afford an extra byte for robustness if you really care.   

You should temper the above with what others have said : If this is hardwired and local, the chances of a bit error is minimal at 9600bps, so maybe you really don't need the overhead.  I'm more of a better safe than sorry person myself. 

If you really don't want the overhead of a CRC, but still want some basic checksum, a really simple checksum is to just XOR each byte with the sum of each previous one.   So for each message you start at '0' in your checksum value, and then for each character you do the following:

checksum^=ch;

Instead sending it in HEX format, do u want me send it in ASCII format ?.If so, it will definitely increase payload size right?.As the Application is timing critical, decoding all the packets back to Hex at PC side and Encoding it in TX side for every transmission will be a big challeng for me.
How about appending Length of payload after HEADER?

It is almost always safer to not encode length, but instead some sort of framing indicator.  If you're sending this to a PC, I sure would be tempted to just encode the whole thing in ASCII, and use CRLF as the terminator, so you get strings on the PC which look like:

ff23723af342473f237223a32234ff23723af342473f237223a32234

The advantage of doing this is that it is really easy to separate out messages on the PC -  you're just looking for hexadecimal characters followed by a 0x0d 0x0a.   

Once you've sorted out how to detect a complete message, it is ok to embed frame lengths as part of the message..  But the over the wire protocol should not rely on length to detect the start and and of a frame.

As an aside ... have you been able to get the hardware-based PIC18 CRC to work?   I've struggled with it for several days on and off, and never got it to work;  The register that stores the CRC is always wrong in my case.   I followed every word of the user guide/data sheet on the hardware CRC, checked the eratta ... no luck.  I ended up going with the table-based CRC + algorithm and it worked correctly the first time.     Is there a general trick to using the hardware CRC?

[magetoo]

What kind of "correctness" do you need?  What is the worst case scenario if a faulty packet is received?  If the worst that can happen is something like a heater turning on when it shouldn't for one second, wasting a miniscule amount of electricity, then don't bother with anything fancy.  The next packet received will fix it.

But if you really need near-perfect reliability, then do look into error-correcting codes.  I would suggest the book A Commonsense Approach to the Theory of Error Correcting Codes by Benjamin Arazi (MIT Press, 1988; ISBN 0-262-01098-4), it's relatively free of higher math and shows how to build CRC generators, Hamming and Reed-Solomon encoders and decoders using just shift registers and XOR gates.  (I haven't written any code, but I don't see how a few bit shifts and XOR operations can be that taxing for even a low-end micro.)

Turns out a lot of this stuff can be fairly simple in practice, it's just the explanations and jargon that is overly complicated.

[cellularmitosis]

I just recently implemented a packet format with CRC8 between an Arduino and a Python script on the PC.  Thought I'd share a specific worked example:

https://github.com/pepaslabs/lab-logger/blob/78ae579e09cad7eea7615533d5952381e60a404a/client-script/Si7021-logger.py

https://github.com/pepaslabs/lab-logger/tree/78ae579e09cad7eea7615533d5952381e60a404a/firmware

[sangar]

Hi Jeroen3,
Can you explain me that how did you calculate CRC7_POLY = 0x91 in the below clode.

Code: [Select]

unsigned char crc8(unsigned char message[], unsigned char length)
{
    const unsigned char CRC7_POLY = 0x91;
    unsigned char i, j, crc = 0;
    for (i = 0; i < length; i++){
        crc ^= message[i];
        for (j = 0; j < 8; j++)
        {
            if (crc & 1)
                crc ^= CRC7_POLY;
            crc >>= 1;
        }
    }
    return crc;
}


Regards,
Muthu

[Jeroen3]

Some code you write steal once.

[TomS_]

"COBS transforms a data set of up to 254 bytes..."

254 contiguous non-zero bytes... specifically.

With the right byte pattern, the length can be arbitrary as stated. All you need is a zero byte within every 254 bytes to reset the pointer to the next "interpreted byte" and you can go on and on for as long as you like.

Ive only just discovered COBS through this thread, and it looks really neat. Im now considering implementing it in to a project I am working on.

edit: or as voltsandjolts says, the 0xFF "special case" (still learning about it!)

[voltsandjolts]

With the right byte pattern, the length can be arbitrary as stated. All you need is a zero byte within every 254 bytes to reset the pointer to the next "interpreted byte" and you can go on and on for as long as you like.

No, you don't need a zero byte. The input data can be any stream of bytes.

[TomS_]

No, you don't need a zero byte. The input data can be any stream of bytes.

Yeah that was based on an initial understanding, but as I read more and more I understood more and more.

I left everything in with an edit at the end because Im not trying to hide that.