Portable Safe Way To Send Struct Between 2 Microcontrollers?

[hal9001]

Hello. I want to send a structure with UART from one microcontroller to another micro. They have different architectures. I experience problems because of the different endianess and padding between the two micros. How is the best way to send this data between 2 micros in a portable way to run on any architecture?


Cheers!

[woofy]

Have you seen this thread:
https://www.eevblog.com/forum/microcontrollers/how-to-prevent-the-compiler-messing-with-a-structures-allignment/msg5025697/#msg5025697

[nctnico]

Hello. I want to send a structure with UART from one microcontroller to another micro. They have different architectures. I experience problems because of the different endianess and padding between the two micros. How is the best way to send this data between 2 micros in a portable way to run on any architecture?

You already discovered the worst way: transferring packed structs verbatim and map a structure onto the buffer. That is a recipy for a dissaster due to endianess and alignment requirements. It is a recurring topic on this forum.

The way that always works is to create accessor functions that get / put values (8 bit wide, 16 bit wide, 32 bit wide, multi-byte fields, etc) from / into a buffer sequentially. These accessor functions use bit shifts to shift the bytes so you can use unpacked, properly aligned (for the platform) structs at both ends to store the data. When getting data, it is also a good idea to do range checks so bad data doesn't screw your system up.

But have you considered sending the information as text? Look at the NMEA format used by GPS receivers for an example. In many situations using text is not longer compared to transferring binary. The big benefits are that 1) you can make the protocol backward and forward compatible 2) as text is human readable, the system will be easier to debug.

I have designed & implemented lots of protocols and the binary protocols where always the most cumbersome ones to implement and debug because they typically change between versions and need extra tools to decipher. With text, you can add / delete fields on one side of the system while the other side keeps on running. This makes it much easier to have different versions in the field because they will still work together for as long as the required feature set overlaps.

[tggzzz]

Hello. I want to send a structure with UART from one microcontroller to another micro. They have different architectures. I experience problems because of the different endianess and padding between the two micros. How is the best way to send this data between 2 micros in a portable way to run on any architecture?

Why do you want to send a sequence of semi-arbitrary bits which can be interpreted as containing specific information? Particularly since a different platform will probably interpret them differently.

OTOH I can understand wanting to send the information represented by the bits.

That implies you must design a means of representing the information that is known by both sender and receiver. There are many such standard mechanisms, e.g. ASN.1, CORBA, JSON, XML, but those may be too heavyweight for your requirements.

[pqass]

Quick and dirty way is to create two functions, char *marshal_message_x(struct *), and its inverse, struct *unmarshal_message_x(char *).
Use sprintf() and sscanf() along with additional info like lengths, type identifiers, and CRC for integrity.

Once upon a time, one standard way was to use XDR.  See: Tutorial, RFC, library.   
I suspect this may be too advanced for you but read the Overview in the tutorial to apprieciate the problem.

[betocool]

I guess it depends on the amount of data.

If it's struct with a handful of bytes, ints, floats, etc, my personal preference is protocol buffers (protobufs). There is a nice implementation for microcontrollers, nanopb, and there are encoders and decoders for (almost) every language you can think of.

More data, in the 100K+ range, I tend to do binary files.

There are other protocols, like JSON, but I'm not a fan of doing string conversions and comparisons on a micro. But that's just me, they work just as well.

Cheers,

Alberto

[uer166]

Out of curiosity what micro do you have that is big endian?

[krho]

Why not use CBOR, which AFAIR has a multiple implementations for micro-controller tinyCBOR from Intel for example is maintained. Instead of something text based which can be hard to parse on the MCU side. Another alternative as said is also protobuf, if you'd ever want to communicate with the PC or android device via Bluetooth where you can easily use the proto definitions to create clients for your language.

[tellurium]

This makes JSON printing / parsing trivial: https://github.com/cesanta/str

The parser is a single-pass FSM, so it is fast. Would be curious to see a benchmark against e.g. protobuf conversion.

Disclaimer: I work for a company that owns that code.

[PlainName]

This makes JSON printing / parsing trivial: https://github.com/cesanta/str

The parser is a single-pass FSM, so it is fast. Would be curious to see a benchmark against e.g. protobuf conversion.

Am I wrong in thinking that the way it works is that you search the incoming data for a JSON identifier, giving the expected type and storage info in which to place it? If so, I am not sure that's a good way to do things - it is fine for a few items, but if you have many then you are searching for every one that you know about, and they may not exist in the data. I would have thought a better way would be to get an identifier and then look up if you know about it, then you can request the data for it. That way you only lookup what's actually there.

[DavidAlfa]

JSON  is overkill for data exchange between two mcus, I don't see the point?
Just used packed attribute and a magic number to determine endianness, for example uint16_t magic=0x1234.
If the receiver gets 0x1234, no action required, if it sees 0x4321 then any variable larger than a byte need byte order swapping.
This will need a small function at the receiver side.

If padding/alignment is not guaranteed due different compilers, you will need other method, like sending each variable separately.

This could be as simple as:

[START]

8bit:
[STRUCT ELEMENT ID][DATA]

16bit:
[STRUCT ELEMENT ID][DATA>>8 & 0xFF][DATA & 0xFF]

32bit:
[STRUCT ELEMENT ID][DATA>>24 & 0xFF][DATA>>16 & 0xFF][DATA>>8 & 0xFF][DATA & 0xFF]

[END]

The struct variable ID should determine the size to be sent/received, so no data length is required.

[abyrvalg]

General questions -> general answers. Does OP want to code a lib for arbitrary communications between <insert any> and <insert any> MCUs? Or just needs to send 3 bytes from a specific PowerPC to a specific Atmega and nothing more? The desired portability level would be quite different.

[Siwastaja]

Endianness swap macros can be used when accessing struct members, but that pretty much ruins the elegancy in using data structure to describe the layout and now you are combining the worst features of access-by-program-code and access-by-data-structure approaches, and can as well do what nctnico suggests. I use (packed) structs when I want fixed-length binary communication between two endpoints where endianness is fixed by design (so practically LE) because it simplifies program code and reduces risk of human errors, satisfying the "don't repeat yourself" and "describe data as data, not code" patterns, but it isn't an universal solution to everything.

Just don't make the mistake of copy-pasting bitshifting stuff. Write the access helpers (functions or macros), again, DRY.

[DavidAlfa]

How about specifying the endianness? I guess the compiler will know it's little endian at compile time and make everything for us?

Code: [Select]

struct __attribute__((packed, scalar_storage_order("little-endian")))
This example effectively changes the data order, and still seen normally by the program:

Code: [Select]


typedef struct __attribute__((packed, scalar_storage_order("little-endian"))) {                  // Or big-endian
  uint8_t a;
  uint16_t b;
  uint32_t c;
} data_t;

volatile data_t data = {
    .a = 0xAA,
    .b = 0x55CC,
    .c = 0x12345678
};

Big endian disassembly:

Code: [Select]

if(data.c!=0x12345678)
    __NOP();

 80007ae: 78c3      ldrb r3, [r0, #3]
 80007b0: 7901      ldrb r1, [r0, #4]
 80007b2: 7942      ldrb r2, [r0, #5]
 80007b4: 0209      lsls r1, r1, #8
 80007b6: 4319      orrs r1, r3
 80007b8: 0412      lsls r2, r2, #16
 80007ba: 7983      ldrb r3, [r0, #6]
 80007bc: 4311      orrs r1, r2
 80007be: 4a32      ldr r2, [pc, #200] ; (8000888 <main+0xe0>)
 80007c0: 061b      lsls r3, r3, #24
 80007c2: 430b      orrs r3, r1
 80007c4: 4293      cmp r3, r2
 80007c6: d000      beq.n 80007ca <main+0x22>
 80007c8: 46c0      nop ; (mov r8, r8)

Little endian disassembly:

Code: [Select]

  if(data.c!=0x12345678)
    __NOP();

 80007ae: 78c1      ldrb r1, [r0, #3]
 80007b0: 7902      ldrb r2, [r0, #4]
 80007b2: 7943      ldrb r3, [r0, #5]
 80007b4: 0412      lsls r2, r2, #16
 80007b6: 0609      lsls r1, r1, #24
 80007b8: 4311      orrs r1, r2
 80007ba: 021b      lsls r3, r3, #8
 80007bc: 7982      ldrb r2, [r0, #6]
 80007be: 430b      orrs r3, r1
 80007c0: 4313      orrs r3, r2
 80007c2: 4a31      ldr r2, [pc, #196] ; (8000888 <main+0xe0>)
 80007c4: 4293      cmp r3, r2
 80007c6: d000      beq.n 80007ca <main+0x22>
 80007c8: 46c0      nop ; (mov r8, r8)





So now it gets as simple as sending/receiving (uint8_t*)data, sizeof(data).

[PlainName]

struct __attribute__((packed, scalar_storage_order("little-endian")))

I appreciate gcc is the Google of compilers, but there are other games in town and making something dependent on a specific tool isn't really any better than the problem it's solving.

[DavidAlfa]

Of course, this will only work if you're already using GCC for all MCUs!

[AVI-crak]

I want to send a structure with UART from one microcontroller to another micro.

The UART is designed to work with text. As soon as a zero is encountered, it will signal the built-in end-of-line processing. To get around the zero problem - you will need to rewrite the entire low-level UART implementation, including changes to the hardware.
In addition there will be a lot of complexity with packet synchronization. Yes, your data structure is no longer a structure, but a packet - it has a beginning and an end.
There is an option to translate all numbers of the structure into HEX format. In this case synchronization by key characters is available - HEX format is quite redundant. Besides, this is the fastest option of program encoding/decoding. Everything else is much more complicated.

In order not to reinvent the wheel - it is easier to use CAN, even without transmitters (if the target is close). CAN does all the low level processing by itself, without your attention. Every microcontroller has CAN now, it's stupid not to use it. 

[nctnico]

I want to send a structure with UART from one microcontroller to another micro.

The UART is designed to work with text. As soon as a zero is encountered, it will signal the built-in end-of-line processing. To get around the zero problem - you will need to rewrite the entire low-level UART implementation, including changes to the hardware.

No, not at all! A UART will send whatever data is fed into the transmit register. Including 0. There is no limitation in the hardware where it comes to what values can be send or received; it is limited only by the configured number of bits (unless it is buggy). Perhaps you are misguided by some high-level libraries that don't allow sending zero due to a bug or a design choice.

[langwadt]

I want to send a structure with UART from one microcontroller to another micro.

The UART is designed to work with text. As soon as a zero is encountered, it will signal the built-in end-of-line processing. To get around the zero problem - you will need to rewrite the entire low-level UART implementation, including changes to the hardware.

where on earth did you get that wrong idea?

In order not to reinvent the wheel - it is easier to use CAN, even without transmitters (if the target is close). CAN does all the low level processing by itself, without your attention. Every microcontroller has CAN now, it's stupid not to use it.

not every MCU have CAN, CAN is substantially more complicated than stuffing just bytes into a uart, and classic CAN is limited to 8 bytes per frame so you have to invent a protocol for longer messages

[AVI-crak]

UART is an old primitive interface, and some implementations in hardware can have very mediocre performance.
When transmitting zero - loss of frame synchronization is possible.

[nctnico]

UART is an old primitive interface, and some implementations in hardware can have very mediocre performance.
When transmitting zero - loss of frame synchronization is possible.

No, because the start and stop bits have a different polarity.

[AVI-crak]

Judging by the topic header, we are talking about two microcontrollers in the cheapest possible version - without quartz resonators.
Transmission speed is the highest, frequencies of microcontrollers are the highest. The cheapest PCB design, maybe even long interconnect wires.
In such conditions, zero transfer via UART is the last drop to a full barrel of problems.
Yes, I had a project like that, I had to abandon the UART.

[DavidAlfa]

Someone is mistakenly thinking C's null string termination applies to the UART?
Because it doesn't! Zero is just data. Start, "0", stop.
No difference in integrity between sending 0, 1, 0x35, etc, it's the same thing unless you don't know how to properly use it.
Who said cheapest layout, fast MCUs, long wires? Don't start again with fictional facts.

Also, OP evaporated completely.

[langwadt]

Judging by the topic header, we are talking about two microcontrollers in the cheapest possible version - without quartz resonators.
Transmission speed is the highest, frequencies of microcontrollers are the highest. The cheapest PCB design, maybe even long interconnect wires.
In such conditions, zero transfer via UART is the last drop to a full barrel of problems.
Yes, I had a project like that, I had to abandon the UART.

I don't know what you managed to screw up but sending zero with a UART was most definitely not the cause of the problem 

[artag]

Convert the data to a known byte-order at each end and back to the local byte-order at the other end.
There are calls in the C library to do this - you don't need to know what each end does. Note that at one end no work will be done because if the endianness differs one will already be network byte order.
These calls are used in all TCP networking code. You can be sure they will be heavily optimised.
https://stackoverflow.com/questions/36924598/understanding-htonl-and-ntohl

A separate problem is delimiting the records and ensuring data transparency. Typically a procedure called byte-stuffing is used to reserve a few bytes for record management and ensure they will never be sent as data. To do this, it 'escapes' teh reserved bytes by prefixing them with another special byte and sending a modified value.
Finally, use a checksum or CRC to catch any corrupted values.

Some people think this is overkill and use a 'simpler' scheme - often a byte count followed by the bytes of the message. Do not do this. It is foolish. Do it properly, or suffer the 1 byte in 10,000 that gets corrupted and needs the system to be reset before it can recover.

(this is broadly agreeing with nctnico. Converting everything to text is an alternative that may be easier if you don't have much data to send)

[nctnico]

Convert the data to a known byte-order at each end and back to the local byte-order at the other end.
There are calls in the C library to do this - you don't need to know what each end does. Note that at one end no work will be done because if the endianness differs one will already be network byte order.
These calls are used in all TCP networking code. You can be sure they will be heavily optimised.
https://stackoverflow.com/questions/36924598/understanding-htonl-and-ntohl

These are not suitable as these functions don't work on byte buffers. You'll need similar functions (or macros) which can work with a byte buffer as input or output. The compiler will optimise such functions as far as the platform allows.

I second the suggestion for having checksums over the data.

[SiliconWizard]

Convert the data to a known byte-order at each end and back to the local byte-order at the other end.
There are calls in the C library to do this - you don't need to know what each end does. Note that at one end no work will be done because if the endianness differs one will already be network byte order.
These calls are used in all TCP networking code. You can be sure they will be heavily optimised.
https://stackoverflow.com/questions/36924598/understanding-htonl-and-ntohl

Uh. This is not the standard C library, it's POSIX stuff. POSIX is not the standard C library.
And the probability you have access to POSIX network functions on microcontrollers is close to the absolute zero in general.

The OP's question was "between 2 microcontrollers".
Besides, these functions "convert values between host and network byte order", whatever the "network byte order" means on a given platform supporting POSIX.
I fail to see how that could answer the OP's question here.

Handling byte swap for handling different endianness is pretty basic stuff, it's not rocket science.
The issue is usually not implementation, it's documentation.

[tellurium]

Not sure if that will help the OP, but I guess I'll mention that.
We had a project like OP mentioned - two MCUs, they had to communicate via the UART.
In the end, the solution was - a text-based communication: JSON messages separated by new line characters, with crc32 followed each JSON string. Something like that:

Code: [Select]

{"method":"ble.adv", "params":{....}} 0x12345678 \n
{"id":1, "method":"fs.list"} 0x34567890 \n
{"id":1, "result": ["a.txt", "b.txt"]} 0xaabbccdd \n

I've no clue how big the anticipated data is, what's the expected data exchange rate, etc etc.
The solution above might be feasible for the moderate, or low, data exchange rate. It is human readable, so it is easy to debug/test.

[rhodges]

The UART is designed to work with text. As soon as a zero is encountered, it will signal the built-in end-of-line processing.

NO. I can't use "thumbs down" on your message, so I will approve the others who show you are wrong.

Did you say this before? If so, please stop.

[Fredderic]

As everyone's been saying, byte-swapping is not hard!  Decide on a byte order, document it, and write functions that copy the data between the internal structure and a transmission buffer.  I'd additionally include a structure ID tag and length on the front, plus a CRC on the end as has also been noted (also presumes some kind of error recovery).  It's a little tedious and possibly error prone perhaps, but it's also simple and reliable, and that's what testing is for.  Best of all, it completely removes both endianess and padding concerns since you're directly controlling the wire format.

That seems just basically sensible to me.  Those send and receive functions also allow you to reorder the internal structure fields, adjust padding, even allow for different machine word sizes (as long as you're using something big enough for the data), and allow the compiler to freely do the same, without disrupting the wire format.  It's also quite easy once you have a few, to toss together a little Python to generate those functions out of a JSON schema (they don't need to be pretty), and the schema in turn makes up part of the documentation.

Annoyingly, most of the "standard" formats seem to either go for minimal binary representation, or maximal flexibility at the expense of size, I haven't seen one I'd say is focused on practicality, which is what I'm generally aiming for; knowing how much memory you'll need as soon as possible, not having to parse a structure you don't need to deal with (important in bus situations), and being able to sit and wait until there's enough space in the outbound buffer (or the entire structure has been received) to avoid double-copying or more convoluted processes, etc.

I generally aim to use the endieness of the smallest processor — so it has the least work to do — and to front-load as much of the fixed-size data as possible.  And then for any more flexible fields, I substitute their size into the transmission structure, and attach the variable part(s) to the end (can sometimes omit the size of the last variable part, thanks to having the overall packet size up front).  A small helper function per structure that computes the space it'll take up in the transmission stream also comes in particularly useful; both for prefixing to the data, and making sure there's sufficient space in the transmit buffer.

Personally, I also rather like Protocol Buffers' flexible integer format (comes in rather handy at times — especially for the packet length header field), though they do break the optimisation of having all your fixed data up front.  But it's clean, easy, and more efficient than hex or b64.  (I've also seen UTF-8 encoding hijacked for sending numbers…  it works…  mostly).

Just remember to document all the things, especially if you're eschewing a standard format for something more pragmatic.

[hans]

The UART is designed to work with text. As soon as a zero is encountered, it will signal the built-in end-of-line processing.

NO. I can't use "thumbs down" on your message, so I will approve the others who show you are wrong.

Did you say this before? If so, please stop.

Indeed, it couldn't be farther from the truth. If a zero byte (or any other byte with enough consecutive zeros) is causing framing errors, then that setup is broken. If that is the UARTs fault, then that is someones incompetence.
Obviously its useful to be aware how an UART works, and why e.g. having devices power cycle can cause (recoverable) faults.

Wrt to solution posted by DavidAlfa: I'm not fully into the big endian scheme.. but if I read about older ARM systems having a different byte order for floating point (its not standardized in IEEE754), or different big endian schemes such as be32 and be8 (thus causing bytes to be swapped or not depending on read size)... There can also be memory alignment issues if casting a buffer to some structure pointer, as well. Personally I wouldn't approach it like that anymore.

Its not that hard to write a stream serializer and deserializer. You initialize it with a (circular)buffer and a start index, and then expose readUint8(), readFloat(), etc. functions. You can do all the platform specific implementations in there you ever need. I'm not sure if speed is that much of a concern here, unless one is really trying to save the final cycle on their CPU.
Json, Bson or protobuf is also an option. But it typically drags in a library dependency. For transmission, Json is not that hard.. but reading is usually the more fiddly part. The last time I did that was on Arduino, which had some convenient libraries to handle that. A larger MCU like ESP32 can easily run those functions.

[voltsandjolts]

In addition there will be a lot of complexity with packet synchronization. Yes, your data structure is no longer a structure, but a packet - it has a beginning and an end.

Data structure and packet framing are best treated as separate issues, and thus separate layers of your communication stack, whatever that may be. COBS is an efficient packet framing method which places no restrictions on data content, making for a neat self-contained layer of the stack.

[Siwastaja]

OP went silent, but out of pure interest, I re-ask the question, what is the big-endian microcontroller?

[AVI-crak]

The forward order (from junior to senior) is used on almost all modern architectures. However, the reverse order is used on an impressive number of types of communication interfaces, as well as on a variety of data storage formats.
For example, everything related to data storages uses reverse data order in the description of data structures. Just like the very first punched cards.
A lot of time has passed, a lot of operations are completely automatic at the hardware level. This gives the illusion of external simplicity of data handling.
Until someone goes down to the bottom of the basement.

[hal9001]

Thanks all 

How about specifying the endianness? I guess the compiler will know it's little endian at compile time and make everything for us?

Code: [Select]

struct __attribute__((packed, scalar_storage_order("little-endian")))

This work, many thanks.

struct __attribute__((packed, scalar_storage_order("little-endian")))

I appreciate gcc is the Google of compilers, but there are other games in town and making something dependent on a specific tool isn't really any better than the problem it's solving.

Maybe not a very portable solution still 

Judging by the topic header, we are talking about two microcontrollers in the cheapest possible version - without quartz resonators.
Transmission speed is the highest, frequencies of microcontrollers are the highest. The cheapest PCB design, maybe even long interconnect wires.
In such conditions, zero transfer via UART is the last drop to a full barrel of problems.
Yes, I had a project like that, I had to abandon the UART.

Uart baud rate is 57600. No corruption of uart data, problem is the struct members dont align with the right bytes.

OP went silent, but out of pure interest, I re-ask the question, what is the big-endian microcontroller?

Turns out both micros are both little endian but one is 8 bit the other is 32. The problem is the padding. Im still after a solution that is portable in case I move to another micro thats big endian. Is big endian not so common I shouldnt worry about it?

[Siwastaja]

Turns out both micros are both little endian but one is 8 bit the other is 32. The problem is the padding. Im still after a solution that is portable in case I move to another micro thats big endian. Is big endian not so common I shouldnt worry about it?

While DavidAlfa's endianness attribute would be a little bit "weird" beast, at least packed attribute/pragma is available on any compiler worth using, so even if still not standard C, it's pretty damn common. And despite some claims on this very forum, I have never seen or heard any compiler generating unaligned accesses using a packed struct, but of course being non-standard, it's not guaranteed portable, even if it practically is.

However, if you can freely design the structure, I strongly suggest to make the members naturally-aligned for the 32-bit target (it will obviously be aligned for the 8-bitter, too). Natural alignment generates smallest and fastest code. (Unaligned members need more instructions, which compiler automagically generates for packed structs, or which you manually write using the nctnico's way, with bitshifts etc.)

Natural alignment means that 32-bit members start at 4-byte boundaries, 16-bit members at 2-byte boundaries etc.. You get closest to this (minimum padding required) by simply ordering the members from biggest to smallest (ages old trick). Then, if padding is still needed for natural alignment, you can pad members manually - this has added benefit of it becoming documented, and giving you explicit reserved members you can later start using when extending the protocol. It's also more obvious to zero such explicit padding members (uninitialized compiler-generated padding is a semi-common safety bug). Add a comment to reserved members such as "always write to 0". Then any non-zero value would denote you are using the extended feature.

When you manually pad the struct to natural alignment, compiler has no reason to add more padding, and thus a packed attribute becomes nearly irrelevant (however, see the important point below). It's a good idea to add a few static_asserts() to verify the offsetof() of a few members, and maybe sizeof of your struct. Then any false assumption on your behalf would simply result in an error.

Remember, if you pointer-cast your struct type over a randomly aligned byte buffer, it will only work if the struct type has the packed attribute. If you ever plan doing that, the options are packed structs and restricting yourself to compilers which behave this way (so practically anything sane), or creating a temporary struct object and memcpying to it, or using accessor helper functions which internally use bitshifts, anding and orring.

[DavidAlfa]

How about generating a constant table containing each struct element size?
Then the sending fucntion only need to check this, and swap the bytes or now depednign on the system endianness.
The receiving function would do the same thing.

Code: [Select]

#if !defined(__BYTE_ORDER__) ||  !defined (__ORDER_LITTLE_ENDIAN__) || !defined  (__ORDER_BIG_ENDIAN__)
    #error "__BYTE_ORDER__  not supported by this compiler!"
#endif


struct __attribute__((packed)) typedef struct {
  uint8_t a;
  uint16_t b;
  uint32_t c;
} data_t;

data_t data;

const uint8_t data_sizes[] = {
  sizeof(data.a);
  sizeof(data.b);
  sizeof(data.c);
}

void send_data(void){
  uint8_t * d = (uint8_t *) data;
  for(uint8_t i=0;i<sizeof(data_sizes);i++){
    if(data_sizes[i] == 1){
      Uart_tx(*d); d++;
    }   
   
#if(__BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__)
    else if(data_sizes[i] == 2){
      Uart_tx(*d++);
      Uart_tx(*d++);
    }   
    else if(data_sizes[i] == 4){
      Uart_tx(*d++);
      Uart_tx(*d++);
      Uart_tx(*d++);
      Uart_tx(*d++);
    }
   
#else
    else if(data_sizes[i] == 2){
      Uart_tx(*(d+1));
      Uart_tx(*d);
      d+=2;
    }   
    else if(data_sizes[i] == 4){
      Uart_tx(*(d+3));
      Uart_tx(*(d+2));
      Uart_tx(*(d+1));
      Uart_tx(*d);
      d+=4;
    }
    #endif   
  }
}


[uer166]

Is big endian not so common I shouldnt worry about it?

Yeah pretty much, unless you can name a specific MCU that you know you might use. It would certainly be a special snowflake, I just wouldn't worry about it at all. Thanks for answering the endianness question because big endian is just not common.

[NorthGuy]

This is not how it works. You do not start from writing a C code. You start from designing a protocol. UART transfers byte streams. You need a method to embed structures (packets) in these stream, perhaps check integrity (or not), perhaps use packet numbering (or not). You need to find a way to find these structures in the UART stream on the receiving end. You design a protocol which can do all of that.

Only then you write the C code which implements the protocol on one MCU (you will need two instances of it of course) and you test it, perhaps revise your protocol in process. Now you have a working communication.

If your goal is to port it to other MCU(s), you try to make it portable from the very beginning. You design your protocol so that you wouldn't have alignment problems, wouldn't depend on idiosyncrasies of your particular MCU, would be easy to compose and parse. You write your C code accordingly. Then, by the time you have created a working model with two MCUs, your code is already portable. When you need to port to a different MCU, you simply use the same code. You probably need to replace functions which send/receive bytes, and you're done. Porting was easy because you thought of that from the very beginning, when you started designing your protocol.

[pqass]

A separate problem is delimiting the records and ensuring data transparency. Typically a procedure called byte-stuffing is used to reserve a few bytes for record management and ensure they will never be sent as data. To do this, it 'escapes' teh reserved bytes by prefixing them with another special byte and sending a modified value.
Finally, use a checksum or CRC to catch any corrupted values.

Some people think this is overkill and use a 'simpler' scheme - often a byte count followed by the bytes of the message. Do not do this. It is foolish. Do it properly, or suffer the 1 byte in 10,000 that gets corrupted and needs the system to be reset before it can recover.

Fortunately with UARTs you can send a BREAK as an end-of-message indicator so byte-stuffing won't be necessary.

If your transmitter doesn't directly support it, you can just lower the baud rate temporarily and send a zero byte.  The receiver will see that as a zero byte with a framing error, that is, the stop bit wasn't detected when expected.

The sender could send a binary or human-readable message consisting of the following fields: 
           magic number, CRC, message sequence, type, version, marshaled structure, BREAK

The receiver can loop reading all characters into a buffer^*.  Only after a BREAK is received will it validate the buffer against the sent CRC and if they match, the buffer can be processed.  Another message can be sent back to report success or error.

Any noise before the magic number can be ignored. If the magic number is found but CRC doesn't match then a error message can be returned, otherwise, the buffer is discarded without reply. You don't want to generate a reply storm on a noisy line.  The sender would just re-send if it didn't get any reply.

^* circular buffer holding the last n characters where n > largest message.

[PlainName]

This is not how it works. You do not start from writing a C code. You start from designing a protocol. UART transfers byte streams. You need a method to embed structures (packets) in these stream, perhaps check integrity (or not), perhaps use packet numbering (or not). You need to find a way to find these structures in the UART stream on the receiving end. You design a protocol which can do all of that.

Only then you write the C code which implements the protocol on one MCU (you will need two instances of it of course) and you test it, perhaps revise your protocol in process. Now you have a working communication.

If your goal is to port it to other MCU(s), you try to make it portable from the very beginning. You design your protocol so that you wouldn't have alignment problems, wouldn't depend on idiosyncrasies of your particular MCU, would be easy to compose and parse. You write your C code accordingly. Then, by the time you have created a working model with two MCUs, your code is already portable. When you need to port to a different MCU, you simply use the same code. You probably need to replace functions which send/receive bytes, and you're done. Porting was easy because you thought of that from the very beginning, when you started designing your protocol.

I don't usually quote whole posts, but this is worth repeating because it's The Way To Do It.

[tggzzz]

This is not how it works. You do not start from writing a C code. You start from designing a protocol. UART transfers byte streams. You need a method to embed structures (packets) in these stream, perhaps check integrity (or not), perhaps use packet numbering (or not). You need to find a way to find these structures in the UART stream on the receiving end. You design a protocol which can do all of that.

Only then you write the C code which implements the protocol on one MCU (you will need two instances of it of course) and you test it, perhaps revise your protocol in process. Now you have a working communication.

If your goal is to port it to other MCU(s), you try to make it portable from the very beginning. You design your protocol so that you wouldn't have alignment problems, wouldn't depend on idiosyncrasies of your particular MCU, would be easy to compose and parse. You write your C code accordingly. Then, by the time you have created a working model with two MCUs, your code is already portable. When you need to port to a different MCU, you simply use the same code. You probably need to replace functions which send/receive bytes, and you're done. Porting was easy because you thought of that from the very beginning, when you started designing your protocol.

I don't usually quote whole posts, but this is worth repeating because it's The Way To Do It.

The OP hasn't addressed the issue of why he wants to send one hardware/compller/flags dependent set of bits to a processor with different hardware/compller/flags. That's philosophically and practically wrong-headed.

Sending the information represented by the set of bits is perfectly sensible, of course.

[Siwastaja]

The OP hasn't addressed the issue of why he wants to send one hardware/compller/flags dependent set of bits to a processor with different hardware/compller/flags. That's philosophically and practically wrong-headed.

Sending the information represented by the set of bits is perfectly sensible, of course.

We can guess: OP has specified one of the controllers is a 8-bitter. These are usually very constrained in resources (CPU time, program code memory and RAM), so using a format which is efficient for this 8-bit CPU and requires minimum amount of parsing there (and possibly more parsing at the 32-bit side) seems sane. For an 8-bitter, JSON, for example, is probably out of question.

[peter-h]

To the OP: just snprintf the values to a buffer (buffers starts with "*") send them out, append a special char e.g. "$" to the buffer, and at the other end discard data until you get a "*", then receive until you get a "$", obviously with buffer overflow detection etc, and then sscanf them back in.

All those complicated schemes...

If the CPU is a Z80 running at 4MHz, and no float code, scale the data appropriately and use integers. 32 bit ints are extremely efficient to output and to input.

[fchk]

There is a lightweight way of transferring data between diifferent architectures since 1995: XDR - the External Data Representation. Used in thing like NFS, RPC, ZFS, ...
See https://en.wikipedia.org/wiki/External_Data_Representation
Standard: https://datatracker.ietf.org/doc/html/rfc4506

[tggzzz]

To the OP: just snprintf the values to a buffer (buffers starts with "*") send them out, append a special char e.g. "$" to the buffer, and at the other end discard data until you get a "*", then receive until you get a "$", obviously with buffer overflow detection etc, and then sscanf them back in.

All those complicated schemes...

If the CPU is a Z80 running at 4MHz, and no float code, scale the data appropriately and use integers. 32 bit ints are extremely efficient to output and to input.

To avoid having printf() code in the 8-bit MCU, send each byte as two hex characters. No rocket science required.

[nctnico]

To the OP: just snprintf the values to a buffer (buffers starts with "*") send them out, append a special char e.g. "$" to the buffer, and at the other end discard data until you get a "*", then receive until you get a "$", obviously with buffer overflow detection etc, and then sscanf them back in.

All those complicated schemes...

If the CPU is a Z80 running at 4MHz, and no float code, scale the data appropriately and use integers. 32 bit ints are extremely efficient to output and to input.

To avoid having printf() code in the 8-bit MCU, send each byte as two hex characters. No rocket science required.

Don't avoid snprintf but use a minimalistic one. Which is likely already provided as part of the standard libraries in various levels of functionality versus size. Re-inventing the wheel (which often leads to unmaintainable code) is not necessary.

[Siwastaja]

To avoid having printf() code in the 8-bit MCU, send each byte as two hex characters. No rocket science required.

This has added benefit of being a constant size conversion i.e., it exactly doubles the size of binary data so buffer sizes and offsets to buffers are easy to calculate. You can then use rest of the characters (e.g. space, comma) as delimiters (message start, message end, whatever is needed). Or, for example, whitespace can be ignored by parser and used as human aid.

Indeed, simple raw hexadecimal tends to be the sweet spot between machine and human readability/writability in very resource constrained systems. It might not be fun to look at hexadecimal output at terminal screen, but it sure is possible. Binary format tends to need writing of some helper tools, after which it is both efficient and easy to use, but some more upfront work.

[peter-h]

To avoid having printf() code in the 8-bit MCU, send each byte as two hex characters. No rocket science required.

Even if you use floating point, you don't need to use printf/scanf provided you settle on a format like 1234.567 and with fixed # of digits.

This is trivial to output and equally trivial to receive.

You treat each half as an int and do itoa for output (two of those, with a '.' between) and atoi (again 2x) for input.

Many years ago I was doing an HPGL parser on an 8MHz Z180 on which doing this was literally 1000x faster than scanf (on an IAR C compiler). It is also very easy in assembler but no point unless the micro is something like a 90S1200 but the if it was that, you aren't likely to use C anyway. I had to do this because the data was arriving at 115k baud and the IAR sscanf was about 100ms!

[tggzzz]

To the OP: just snprintf the values to a buffer (buffers starts with "*") send them out, append a special char e.g. "$" to the buffer, and at the other end discard data until you get a "*", then receive until you get a "$", obviously with buffer overflow detection etc, and then sscanf them back in.

All those complicated schemes...

If the CPU is a Z80 running at 4MHz, and no float code, scale the data appropriately and use integers. 32 bit ints are extremely efficient to output and to input.

To avoid having printf() code in the 8-bit MCU, send each byte as two hex characters. No rocket science required.

Don't avoid snprintf but use a minimalistic one. Which is likely already provided as part of the standard libraries in various levels of functionality versus size. Re-inventing the wheel (which often leads to unmaintainable code) is not necessary.

Neutered printf() is an option, but hex chars are very simple/small/fast and may be sufficient. Who knows

[NorthGuy]

I do not see any benefits in converting binary data to ASCII, other than you can see the text on the terminal. You spend efforts and resources on encoding and decoding, you decrease the throughput. What for?

The reason that many half-a-century old protocols were text-based is because people had terminals and they were literally typing the text in and were watching the results. This is not something you need while interlinking two MCUs, is it?

[nctnico]

I do not see any benefits in converting binary data to ASCII, other than you can see the text on the terminal. You spend efforts and resources on encoding and decoding, you decrease the throughput. What for?

The reason that many half-a-century old protocols were text-based is because people had terminals and they were literally typing the text in and were watching the results. This is not something you need while interlinking two MCUs, is it?

It is. Think in terms of NRE. A lot of time & money gets spend on verification and finding problems / reasons why things fail every now and then. Having text based protocols reduces the time needed to debug systems a lot. At the moment I'm consulting on a project which has quite a few communication devices which use different communication interfaces chained up together. Using text based protocols, do emulation of higher level devices locally and plain text debug output helps a lot to verify correct behaviour of parts of the system and the system as a whole.

[tggzzz]

I do not see any benefits in converting binary data to ASCII, other than you can see the text on the terminal. You spend efforts and resources on encoding and decoding, you decrease the throughput. What for?

Ease of implementation.
Ease of debugging.
Ease of  comprehension in a deployed system.
Simple "packet" framing, parsing and error recovery.

Where efficient utilisation of the comms medium is important, it is worth expending more effort avoiding redundant data. But if that isn't an issue, KISS is a useful principle.

[NorthGuy]

It is. Think in terms of NRE. A lot of time & money gets spend on verification and finding problems / reasons why things fail every now and then. Having text based protocols reduces the time needed to debug systems a lot. At the moment I'm consulting on a project which has quite a few communication devices which use different communication interfaces chained up together. Using text based protocols, do emulation of higher level devices locally and plain text debug output helps a lot to verify correct behaviour of parts of the system and the system as a whole.

Of course, you'll spend lots of time and money when you use such archaic methods.

I have a CAN network with several devices which send lots of various messages - perhaps a hundred every second. If I had to look at them all in text format, it would take me days, if not weeks, to find something in such a stream of text. Instead, I have connected a sniffer to the network, which verifies all the messages, select the messages that I need and shows them to me. If there's any problem, it doesn't take long to figure out what's going on. I can use the same sniffer device to run automated tests which can run for hours (or days) - automatically injecting messages that I want, monitoring responses etc.

Similarly, you can sniff UART or anything else (if not too fast), decode, find errors, write selective logs, show on display, store to a file and parse later etc. etc.

[AVI-crak]

I do not see any benefits in converting binary data to ASCII, other than you can see the text on the terminal. You spend efforts and resources on encoding and decoding, you decrease the throughput. What for?

There's always a limit when the simple things just don't work anymore.

[SiliconWizard]

I do not see any benefits in converting binary data to ASCII, other than you can see the text on the terminal. You spend efforts and resources on encoding and decoding, you decrease the throughput. What for?

Not only that, but it actually makes it more likely to contain bugs. Parsing text 100% correctly, fail-safe, without any buffer overrun and other niceties, that's a lot less trivial than properly writing a binary-based protocol and handling it (yes, as long as you have mastered first the basics, like alignment and endianness, which are really basics and way simpler to master than writing a proper parser, really).

The reason that many half-a-century old protocols were text-based is because people had terminals and they were literally typing the text in and were watching the results. This is not something you need while interlinking two MCUs, is it?

Yeah, well others have mentioned that as a feature that can still be useful - in that case, just write your own encoding/decoding tool for debugging and testing, it'll take a couple hours of your time, well invested, and if there's any parsing bug in it, that's no biggie, it won't impact your product. Just your tool.

But that's always interesting to see how perspectives can be different between people doing basically the same job.

[asmi]

whatever the "network byte order" means on a given platform supporting POSIX.

It always means the same thing on all platforms, which is it's raison d'etre.

[PlainName]

Yeah, well others have mentioned that as a feature that can still be useful - in that case, just write your own encoding/decoding tool for debugging and testing, it'll take a couple hours of your time, well invested, and if there's any parsing bug in it, that's no biggie, it won't impact your product. Just your tool.

Trouble with that is you need to know the likely issue you might have to fix ahead of time in order to write a useful tool, so why not just prevent that issue in the first place? Also, it's well known that the writer of bugs isn't usually the best person to check for that bug, so if there's a protocol issue why would a protocol debugger written by the same person not have the same issue?

Finally, the human eye is generally much better than some predefined general purpose tool for spotting tiny clues that may point to something. Don't get me wrong - a protocol debugger can be very useful indeed, but it's not the be-all and end-all. And if you're unlucky (which, in this case, you'd want to be!) you might spend those hours writing this tool that will never be needed. So the temptation is strong to write it when you have a problem, and then it can be targeted at that specific problem. Being able to recognise data, or handle it simply, may remove the need for that.

[tggzzz]

I do not see any benefits in converting binary data to ASCII, other than you can see the text on the terminal. You spend efforts and resources on encoding and decoding, you decrease the throughput. What for?

Not only that, but it actually makes it more likely to contain bugs. Parsing text 100% correctly, fail-safe, without any buffer overrun and other niceties, that's a lot less trivial than properly writing a binary-based protocol and handling it (yes, as long as you have mastered first the basics, like alignment and endianness, which are really basics and way simpler to master than writing a proper parser, really).

Buffer overrun is always an issue in C-like languages, however the information in the struct is encoded.

On a UART getting the "packet" framing delimiters right in the presence of errors can be error prone. Doubly so where the information is binary encoded and hence could mimic the packet framing delimiters.

If someone can't convert 32-bits to/from hex ASCII characters, they shouldn't be writing any code.

[nctnico]

Yeah, well others have mentioned that as a feature that can still be useful - in that case, just write your own encoding/decoding tool for debugging and testing, it'll take a couple hours of your time, well invested, and if there's any parsing bug in it, that's no biggie, it won't impact your product. Just your tool.

Trouble with that is you need to know the likely issue you might have to fix ahead of time in order to write a useful tool, so why not just prevent that issue in the first place? Also, it's well known that the writer of bugs isn't usually the best person to check for that bug, so if there's a protocol issue why would a protocol debugger written by the same person not have the same issue?

Finally, the human eye is generally much better than some predefined general purpose tool for spotting tiny clues that may point to something. Don't get me wrong - a protocol debugger can be very useful indeed, but it's not the be-all and end-all. And if you're unlucky (which, in this case, you'd want to be!) you might spend those hours writing this tool that will never be needed.

IMHO such a tool is massively usefull for debugging and verification purposes. And you can go a step further as well. Over the decades I've implemented a wide variety of protocols and for several of them I have written simulators to test against (partly due to lack of having the real thing available). During testing with the actual system I made sure to update the simulator as well to get a precise 'model'. The advantage of a simulator is that you can do very extensive tests (including introducing faults) in a short time without manual intervention.

[Siwastaja]

Of course, you'll spend lots of time and money when you use such archaic methods.

This is exactly the case. nctnico's preferred way of verification, generating commands on terminal and verifying results by eye, results in maybe 1% test coverage if even that. It indeed is simplest way for the very first "smoke test" (the advantage is in quick start before the code for the "other side" is finished), but it is nearly guaranteed to miss difficult corner cases.

For real, automated (unit and/or integration) testing, binary format works as well as text or better, and because the parser will be simpler (specifically, has less hidden state and uses (almost) exclusively fixed-size buffers of predictable size), it is less likely to contain bugs. It is surprising how buggy text-based parsers can be, people who think text based protocols are "simple" to implement tend to get stuff like line feeds (four different variants, some of which store state between bytes) seriously wrong.

And don't get me wrong, there is no need to be afraid of mistakes. If you want or need text-based protocol, then of course you will implement it, try to be careful, and test it thoroughly. It is entirely possible to make it completely bug-free. But thinking that this is somehow easier is so blatantly wrong that I would not trust anyone who says that. They are very likely to make bugs.

[nctnico]

Of course, you'll spend lots of time and money when you use such archaic methods.

This is exactly the case. nctnico's preferred way of verification, generating commands on terminal and verifying results by eye, results in maybe 1% test coverage if even that. It indeed is simplest way for the very first "smoke test" (the advantage is in quick start before the code for the "other side" is finished), but it is nearly guaranteed to miss difficult corner cases.

Now you are throwing initial testing and final verification on 1 pile. For text based protocols you can use existing tools (like sed, grep, etc) combined with simple scripts (bash, python, etc) that can parse & check a long log file in text (captured using a terminal emulator) to produce a report. For a binary protocol you'd need put much more effort into tooling for capturing the data and doing the verification / translation. For examples: if you have numbers as columns in text, it is super easy to plot these into a graph using Excel.

[tggzzz]

Of course, you'll spend lots of time and money when you use such archaic methods.

This is exactly the case. nctnico's preferred way of verification, generating commands on terminal and verifying results by eye, results in maybe 1% test coverage if even that. It indeed is simplest way for the very first "smoke test" (the advantage is in quick start before the code for the "other side" is finished), but it is nearly guaranteed to miss difficult corner cases.

For real, automated (unit and/or integration) testing, binary format works as well as text or better, and because the parser will be simpler (specifically, has less hidden state and uses (almost) exclusively fixed-size buffers of predictable size), it is less likely to contain bugs. It is surprising how buggy text-based parsers can be, people who think text based protocols are "simple" to implement tend to get stuff like line feeds (four different variants, some of which store state between bytes) seriously wrong.

And don't get me wrong, there is no need to be afraid of mistakes. If you want or need text-based protocol, then of course you will implement it, try to be careful, and test it thoroughly. It is entirely possible to make it completely bug-free. But thinking that this is somehow easier is so blatantly wrong that I would not trust anyone who says that. They are very likely to make bugs.

Terminal-based testing isn't approriate for unit/integration/regression tests. It is suitable for initial exploration and for "WTF is it doing now" field service tests.

[AndyBeez]

This question is rather like asking, how many lines do I need to draw a polygon? To the OP, I would say first serialise the struct  making sure its instance is serializable. Then buffer the character payload out to the TX/RX lines, using DMA if available. Interconnect the uC via RS232/485/USB, or a 10K resistor, depending on distance. Then scope the traffic with the protocol analyser to test. Other end, do this all in reverse. Then walk away because someone is bound to want to use SOAP over TCP instead. Or I2C.

[SiliconWizard]

Yeah, well others have mentioned that as a feature that can still be useful - in that case, just write your own encoding/decoding tool for debugging and testing, it'll take a couple hours of your time, well invested, and if there's any parsing bug in it, that's no biggie, it won't impact your product. Just your tool.

Trouble with that is you need to know the likely issue you might have to fix ahead of time in order to write a useful tool

No.

[TomS_]

these functions "convert values between host and network byte order", whatever the "network byte order" means on a given platform supporting POSIX.
I fail to see how that could answer the OP's question here.

Network order is not platform dependent, that's the whole point of it.

Network order is specified as big endian.

But using hton() and ntoh() would only be to solve one half of the OPs problem, which would be to provide predictable endianess for the data between the two MCUs. As functions or macros they are easy enough to implement if they are not otherwise available.

How you pack the data is the other half which can be solved another way.

[NorthGuy]

But using hton() and ntoh() would only be to solve one half of the OPs problem, which would be to provide predictable endianess for the data between the two MCUs.

There's a much simple solution to that problem. Use little-endian. There's not 1980 now, and you don't have to worry about endianness when designing new protocols. Look around - there are many store fronts and none of them have anything where people could tie their horses - somehow that is Ok.

[TomS_]

There's a much simple solution to that problem. Use little-endian. There's not 1980 now, and you don't have to worry about endianness when designing new protocols.

Im suggesting ntoh/hton because they exist for the very purpose of making data portable between dissimilar archs.

I dont care for endianess arguments. 

[nctnico]

There's a much simple solution to that problem. Use little-endian. There's not 1980 now, and you don't have to worry about endianness when designing new protocols.

Im suggesting ntoh/hton because they exist for the very purpose of making data portable between dissimilar archs.

As I wrote before: those functions don't help with alignment issues so they only solve half of the problem. You can't use those functions on multibyte variables which are cast from a random point in a byte buffer on all platforms.

[PlainName]

There's a much simple solution to that problem. Use little-endian. There's not 1980 now, and you don't have to worry about endianness when designing new protocols.

Im suggesting ntoh/hton because they exist for the very purpose of making data portable between dissimilar archs.

As I wrote before: those functions don't help with alignment issues so they only solve half of the problem. You can't use those functions on multibyte variables which are cast from a random point in a byte buffer on all platforms.

Not hard to write ones that do work for your particular data. In fact, I'm failing to see why having a decoder (and encoder) is such a problem - if it's not needed it's blank and does nothing, but if they architecture changes and they are needed then there they are! Nothing to fix upstream. Then you're freed from intimately associating your data with your code and, blimey, don't we all know that a high level of coupling is just bad, the information equivalent of spaghetti code. And we also all know that starting with "it's just a quick hack, doesn't need to be thought about" is often the precursor to zombie shit that's buggy as hell but too useful to discard. Not to mention that just getting into the habit of Doing It Right is a big plus by itself.

So here we are preaching the exact opposite of all that, trying to outdo each other in being cheaper and more amateurish. Which would be so bad if all this wasn't aimed at someone wanting to learn something, and guess what he's learning!

[pqass]

I like concrete examples... 
so I decided to create my own implementation of message-passing plumbing.
It may be useful to others as a starting point or just to discuss.

The high-level parts (see below) are: message structures (like struct example_msg), and their corresponding encoder functions (like send_message_example_1()).  Also included are preamble/final envelope helpers.  The intent is to make the code compile/work in both sender and receiver environments (8 bit MCUs to 64 bit CPUs), rather than each having their own independent implementation linked only via documented wire-protocol.

Code: [Select]

#include "msg.h"

char    *_msg_type_example = "EXAMPLE_TYPE";

struct example_msg {
  int errcd;
  char *errmsg;
  float severity;
  char *stack[10];
  int bloblen;
  char *blob;
  int magic[5];
};


void send_message_example_1(int16_t sn, struct example_msg *s) {

  send_message_preamble(sn, _msg_type_example, 1);

  write_int16((int16_t)s->errcd);
  write_asciiz(s->errmsg);
  write_float(s->severity);
  for (int j = 0; j < NELEMS(s->stack); j++)
    write_asciiz(s->stack[j]);
  write_int16((int16_t)s->bloblen);
  write_blob(s->blob, s->bloblen);
  for (int j = 0; j < NELEMS(s->magic); j++)
    write_int16((int16_t)s->magic[j]);

  send_message_final();
}


void send_message_preamble(int16_t sn, char *ty, int16_t vn) {
  _isLittleEndian = isLittleEndian();   // write_xxx() and read_xxx() need this initialized

  write_asciiz(_msg_label); // magic start-of-message label
  crcInit();                // crc starts from here onward
  write_int16(sn);          // sequence number
  write_asciiz(ty);         // message type
  write_int16(vn);          // message version
}

void send_message_final() {
  write_int16((int16_t)crcDone());
  write_break();
}


struct example_msg ex_msg;

void send() {
  ex_msg.errcd=0x1234; ex_msg.errmsg="my errmsg text here"; ex_msg.severity=-9.3;
  ex_msg.stack[0] = "1st";  ex_msg.stack[1] = "2nd";
  ex_msg.stack[2] = "3rd";  ex_msg.stack[3] = "4th";
  ex_msg.stack[4] = "5th";  ex_msg.stack[5] = NULL;
  ex_msg.stack[6] = "7th";  ex_msg.stack[7] = "8th";
  ex_msg.stack[8] = "9th";  ex_msg.stack[9] = "10th";
  ex_msg.blob = "this is a BLOB without trailing zero";
  ex_msg.bloblen = strlen(ex_msg.blob);
  ex_msg.magic[0] = 0xE301; ex_msg.magic[1] = 0xE302;
  ex_msg.magic[2] = 0xE303; ex_msg.magic[3] = 0xE304;
  ex_msg.magic[4] = 0xE305;

  send_message_example_1(0x5678, &ex_msg);
}


Calling send() above will produce the output below.

Notice that null-terminated strings (char *, asciiz) use the \0 as a field delimiter.  Ints just occupy a fixed space (2 bytes) and are in big-endian format. One could change endian-ness or expand to 4, 8 bytes.  I'm not sure if signed/unsigned would need to be accommodated; not too many ones-compliment machines around.  write_float() just prints <whole>.<fraction> as asciiz.  And opaque blobs (which may contain zeros) require a length integer preceeding it. Pointers to/arrays of sub-structs can be described in their own encoder/decoder functions.

Code: [Select]

./mmm s | od -A x -t x1z -v
000000 4d 59 4d 53 47 00 56 78 45 58 41 4d 50 4c 45 5f  >MYMSG.VxEXAMPLE_<
000010 54 59 50 45 00 00 01 12 34 6d 79 20 65 72 72 6d  >TYPE....4my errm<
000020 73 67 20 74 65 78 74 20 68 65 72 65 00 2d 39 2e  >sg text here.-9.<
000030 33 30 30 30 30 30 00 31 73 74 00 32 6e 64 00 33  >300000.1st.2nd.3<
000040 72 64 00 34 74 68 00 35 74 68 00 00 37 74 68 00  >rd.4th.5th..7th.<
000050 38 74 68 00 39 74 68 00 31 30 74 68 00 00 24 74  >8th.9th.10th..$t<
000060 68 69 73 20 69 73 20 61 20 42 4c 4f 42 20 77 69  >his is a BLOB wi<
000070 74 68 6f 75 74 20 74 72 61 69 6c 69 6e 67 20 7a  >thout trailing z<
000080 65 72 6f e3 01 e3 02 e3 03 e3 04 e3 05 00 57     >ero...........W<


The receiver writes all characters as they arrive to a circular buffer (enough space to accommodate the largest message). Once a BREAK signal is detected, the current buffer is then processed. The receiver doesn't know what to expect, except that all messages have a envelope consisting of a magic label to denote start of message, message sequence number, type, version, some amount of fields, then a CRC. Assuming the last two bytes before the BREAK are CRC, the code can recalculate/validate the whole message before it even attempts to decode the envelope or payload fields.  The assumption being that noise can add/change random characters.

The recv_message() function (see below) is called after a BREAK is received to determine which message was sent. It will read and validate the envelope, then call the appropriate decoder function (like recv_message_example_1()) to re-hydrate the original structure.

Code: [Select]

int8_t recv_message(int16_t **snp, char **typ, int16_t **vnp, char **msgp) {
  uint8_t e;

  static int16_t sn;  // allocate sequence number
                      // recv_message_preamble() will allocate for type
  static int16_t vn;  // allocate message version

  *snp = &sn;         // tell caller
                      // recv_message_preamble() will tell caller
  *vnp = &vn;         // tell caller
  *msgp = NULL;       // init in case of early exit

  if ((e = recv_message_preamble(*snp, typ, *vnp)) < 0) return e;

  if (*typ != NULL) {
    if (strcmp(*typ, _msg_type_example) == 0 && vn == 1) {
      static struct example_msg ex_msg;
      if ((e = recv_message_example_1(&ex_msg)) < 0) return e;
      *msgp = (char *)&ex_msg;
    } // else if(strcmp(.....) == 0 && vn == ?) {
  }


  if ((e = recv_message_final()) < 0) return e;

  return SUCCESS;     // structure has been filled
}


int8_t recv_message_example_1(struct example_msg *s) {
  char *p;

  if ((p = read_int16()) == NULL) return EOBUF;
  s->errcd = (int)(*((int16_t *)p));

  static char errmsg[50];
  if ((p = read_asciiz(errmsg, sizeof(errmsg))) == NULL) return EOBUF;
  s->errmsg = p;

  // TODO: finish up the rest

  return SUCCESS;     // structure has been filled
}


int8_t recv_message_preamble(int16_t *sn, char **ty, int16_t *vn) {
  uint8_t e;
  char *p;

  *sn = 0;      // init in case of early exit
  *ty = NULL;   // init in case of early exit
  *vn = 0;      // init in case of early exit
 
  _isLittleEndian = isLittleEndian();   // write_xxx() and read_xxx() need this initialized

  if (!find_asciiz(_msg_label)) return NOMSG;     // no message leader found

//  crcInit();
//  if ((e = peek_validate_crc()) < 0) return e;    // non-destructive read until end of buffer to check crc in last sizeof(CRCTYPE) chars
 
  crcInit();

  // sequence number
  if ((p = read_int16()) == NULL) return EOBUF;
  *sn = *((int16_t *)p);

  // message type
  static char msgty[10];
  if ((p = read_asciiz(msgty, sizeof(msgty))) == NULL) return EOBUF;
  *ty = p;

  // message version
  if ((p = read_int16()) == NULL) return EOBUF;
  *vn = *((int16_t *)p);

  return SUCCESS;
}

int8_t recv_message_final() {
  char *p;

  if ((p = read_int16()) == NULL) return EOBUF;
  uint8_t sent_crc = (uint8_t)(*((int16_t *)p));

  if (sent_crc != crcDone()) return BADCRC;

  return SUCCESS;
}


I haven't finished the receiver part just yet; seg-fault somewhere + unfinished decoder code.
Yeah, I went a bit overboard with double indirction (type **) and static function vars as a way to not use malloc().