Portable Safe Way To Send Struct Between 2 Microcontrollers?

[hal9001]

Hello. I want to send a structure with UART from one microcontroller to another micro. They have different architectures. I experience problems because of the different endianess and padding between the two micros. How is the best way to send this data between 2 micros in a portable way to run on any architecture?


Cheers!

[woofy]

Have you seen this thread:
https://www.eevblog.com/forum/microcontrollers/how-to-prevent-the-compiler-messing-with-a-structures-allignment/msg5025697/#msg5025697

[nctnico]

Hello. I want to send a structure with UART from one microcontroller to another micro. They have different architectures. I experience problems because of the different endianess and padding between the two micros. How is the best way to send this data between 2 micros in a portable way to run on any architecture?

You already discovered the worst way: transferring packed structs verbatim and map a structure onto the buffer. That is a recipy for a dissaster due to endianess and alignment requirements. It is a recurring topic on this forum.

The way that always works is to create accessor functions that get / put values (8 bit wide, 16 bit wide, 32 bit wide, multi-byte fields, etc) from / into a buffer sequentially. These accessor functions use bit shifts to shift the bytes so you can use unpacked, properly aligned (for the platform) structs at both ends to store the data. When getting data, it is also a good idea to do range checks so bad data doesn't screw your system up.

But have you considered sending the information as text? Look at the NMEA format used by GPS receivers for an example. In many situations using text is not longer compared to transferring binary. The big benefits are that 1) you can make the protocol backward and forward compatible 2) as text is human readable, the system will be easier to debug.

I have designed & implemented lots of protocols and the binary protocols where always the most cumbersome ones to implement and debug because they typically change between versions and need extra tools to decipher. With text, you can add / delete fields on one side of the system while the other side keeps on running. This makes it much easier to have different versions in the field because they will still work together for as long as the required feature set overlaps.

[tggzzz]

Hello. I want to send a structure with UART from one microcontroller to another micro. They have different architectures. I experience problems because of the different endianess and padding between the two micros. How is the best way to send this data between 2 micros in a portable way to run on any architecture?

Why do you want to send a sequence of semi-arbitrary bits which can be interpreted as containing specific information? Particularly since a different platform will probably interpret them differently.

OTOH I can understand wanting to send the information represented by the bits.

That implies you must design a means of representing the information that is known by both sender and receiver. There are many such standard mechanisms, e.g. ASN.1, CORBA, JSON, XML, but those may be too heavyweight for your requirements.

[pqass]

Quick and dirty way is to create two functions, char *marshal_message_x(struct *), and its inverse, struct *unmarshal_message_x(char *).
Use sprintf() and sscanf() along with additional info like lengths, type identifiers, and CRC for integrity.

Once upon a time, one standard way was to use XDR.  See: Tutorial, RFC, library.   
I suspect this may be too advanced for you but read the Overview in the tutorial to apprieciate the problem.

[betocool]

I guess it depends on the amount of data.

If it's struct with a handful of bytes, ints, floats, etc, my personal preference is protocol buffers (protobufs). There is a nice implementation for microcontrollers, nanopb, and there are encoders and decoders for (almost) every language you can think of.

More data, in the 100K+ range, I tend to do binary files.

There are other protocols, like JSON, but I'm not a fan of doing string conversions and comparisons on a micro. But that's just me, they work just as well.

Cheers,

Alberto

[uer166]

Out of curiosity what micro do you have that is big endian?

[krho]

Why not use CBOR, which AFAIR has a multiple implementations for micro-controller tinyCBOR from Intel for example is maintained. Instead of something text based which can be hard to parse on the MCU side. Another alternative as said is also protobuf, if you'd ever want to communicate with the PC or android device via Bluetooth where you can easily use the proto definitions to create clients for your language.

[tellurium]

This makes JSON printing / parsing trivial: https://github.com/cesanta/str

The parser is a single-pass FSM, so it is fast. Would be curious to see a benchmark against e.g. protobuf conversion.

Disclaimer: I work for a company that owns that code.

[PlainName]

This makes JSON printing / parsing trivial: https://github.com/cesanta/str

The parser is a single-pass FSM, so it is fast. Would be curious to see a benchmark against e.g. protobuf conversion.

Am I wrong in thinking that the way it works is that you search the incoming data for a JSON identifier, giving the expected type and storage info in which to place it? If so, I am not sure that's a good way to do things - it is fine for a few items, but if you have many then you are searching for every one that you know about, and they may not exist in the data. I would have thought a better way would be to get an identifier and then look up if you know about it, then you can request the data for it. That way you only lookup what's actually there.

[DavidAlfa]

JSON  is overkill for data exchange between two mcus, I don't see the point?
Just used packed attribute and a magic number to determine endianness, for example uint16_t magic=0x1234.
If the receiver gets 0x1234, no action required, if it sees 0x4321 then any variable larger than a byte need byte order swapping.
This will need a small function at the receiver side.

If padding/alignment is not guaranteed due different compilers, you will need other method, like sending each variable separately.

This could be as simple as:

[START]

8bit:
[STRUCT ELEMENT ID][DATA]

16bit:
[STRUCT ELEMENT ID][DATA>>8 & 0xFF][DATA & 0xFF]

32bit:
[STRUCT ELEMENT ID][DATA>>24 & 0xFF][DATA>>16 & 0xFF][DATA>>8 & 0xFF][DATA & 0xFF]

[END]

The struct variable ID should determine the size to be sent/received, so no data length is required.

[abyrvalg]

General questions -> general answers. Does OP want to code a lib for arbitrary communications between <insert any> and <insert any> MCUs? Or just needs to send 3 bytes from a specific PowerPC to a specific Atmega and nothing more? The desired portability level would be quite different.

[Siwastaja]

Endianness swap macros can be used when accessing struct members, but that pretty much ruins the elegancy in using data structure to describe the layout and now you are combining the worst features of access-by-program-code and access-by-data-structure approaches, and can as well do what nctnico suggests. I use (packed) structs when I want fixed-length binary communication between two endpoints where endianness is fixed by design (so practically LE) because it simplifies program code and reduces risk of human errors, satisfying the "don't repeat yourself" and "describe data as data, not code" patterns, but it isn't an universal solution to everything.

Just don't make the mistake of copy-pasting bitshifting stuff. Write the access helpers (functions or macros), again, DRY.

[DavidAlfa]

How about specifying the endianness? I guess the compiler will know it's little endian at compile time and make everything for us?

Code: [Select]

struct __attribute__((packed, scalar_storage_order("little-endian")))
This example effectively changes the data order, and still seen normally by the program:

Code: [Select]


typedef struct __attribute__((packed, scalar_storage_order("little-endian"))) {                  // Or big-endian
  uint8_t a;
  uint16_t b;
  uint32_t c;
} data_t;

volatile data_t data = {
    .a = 0xAA,
    .b = 0x55CC,
    .c = 0x12345678
};

Big endian disassembly:

Code: [Select]

if(data.c!=0x12345678)
    __NOP();

 80007ae: 78c3      ldrb r3, [r0, #3]
 80007b0: 7901      ldrb r1, [r0, #4]
 80007b2: 7942      ldrb r2, [r0, #5]
 80007b4: 0209      lsls r1, r1, #8
 80007b6: 4319      orrs r1, r3
 80007b8: 0412      lsls r2, r2, #16
 80007ba: 7983      ldrb r3, [r0, #6]
 80007bc: 4311      orrs r1, r2
 80007be: 4a32      ldr r2, [pc, #200] ; (8000888 <main+0xe0>)
 80007c0: 061b      lsls r3, r3, #24
 80007c2: 430b      orrs r3, r1
 80007c4: 4293      cmp r3, r2
 80007c6: d000      beq.n 80007ca <main+0x22>
 80007c8: 46c0      nop ; (mov r8, r8)

Little endian disassembly:

Code: [Select]

  if(data.c!=0x12345678)
    __NOP();

 80007ae: 78c1      ldrb r1, [r0, #3]
 80007b0: 7902      ldrb r2, [r0, #4]
 80007b2: 7943      ldrb r3, [r0, #5]
 80007b4: 0412      lsls r2, r2, #16
 80007b6: 0609      lsls r1, r1, #24
 80007b8: 4311      orrs r1, r2
 80007ba: 021b      lsls r3, r3, #8
 80007bc: 7982      ldrb r2, [r0, #6]
 80007be: 430b      orrs r3, r1
 80007c0: 4313      orrs r3, r2
 80007c2: 4a31      ldr r2, [pc, #196] ; (8000888 <main+0xe0>)
 80007c4: 4293      cmp r3, r2
 80007c6: d000      beq.n 80007ca <main+0x22>
 80007c8: 46c0      nop ; (mov r8, r8)





So now it gets as simple as sending/receiving (uint8_t*)data, sizeof(data).

[PlainName]

struct __attribute__((packed, scalar_storage_order("little-endian")))

I appreciate gcc is the Google of compilers, but there are other games in town and making something dependent on a specific tool isn't really any better than the problem it's solving.

[DavidAlfa]

Of course, this will only work if you're already using GCC for all MCUs!

[AVI-crak]

I want to send a structure with UART from one microcontroller to another micro.

The UART is designed to work with text. As soon as a zero is encountered, it will signal the built-in end-of-line processing. To get around the zero problem - you will need to rewrite the entire low-level UART implementation, including changes to the hardware.
In addition there will be a lot of complexity with packet synchronization. Yes, your data structure is no longer a structure, but a packet - it has a beginning and an end.
There is an option to translate all numbers of the structure into HEX format. In this case synchronization by key characters is available - HEX format is quite redundant. Besides, this is the fastest option of program encoding/decoding. Everything else is much more complicated.

In order not to reinvent the wheel - it is easier to use CAN, even without transmitters (if the target is close). CAN does all the low level processing by itself, without your attention. Every microcontroller has CAN now, it's stupid not to use it. 

[nctnico]

I want to send a structure with UART from one microcontroller to another micro.

The UART is designed to work with text. As soon as a zero is encountered, it will signal the built-in end-of-line processing. To get around the zero problem - you will need to rewrite the entire low-level UART implementation, including changes to the hardware.

No, not at all! A UART will send whatever data is fed into the transmit register. Including 0. There is no limitation in the hardware where it comes to what values can be send or received; it is limited only by the configured number of bits (unless it is buggy). Perhaps you are misguided by some high-level libraries that don't allow sending zero due to a bug or a design choice.

[langwadt]

I want to send a structure with UART from one microcontroller to another micro.

The UART is designed to work with text. As soon as a zero is encountered, it will signal the built-in end-of-line processing. To get around the zero problem - you will need to rewrite the entire low-level UART implementation, including changes to the hardware.

where on earth did you get that wrong idea?

In order not to reinvent the wheel - it is easier to use CAN, even without transmitters (if the target is close). CAN does all the low level processing by itself, without your attention. Every microcontroller has CAN now, it's stupid not to use it.

not every MCU have CAN, CAN is substantially more complicated than stuffing just bytes into a uart, and classic CAN is limited to 8 bytes per frame so you have to invent a protocol for longer messages

[AVI-crak]

UART is an old primitive interface, and some implementations in hardware can have very mediocre performance.
When transmitting zero - loss of frame synchronization is possible.

[nctnico]

UART is an old primitive interface, and some implementations in hardware can have very mediocre performance.
When transmitting zero - loss of frame synchronization is possible.

No, because the start and stop bits have a different polarity.

[AVI-crak]

Judging by the topic header, we are talking about two microcontrollers in the cheapest possible version - without quartz resonators.
Transmission speed is the highest, frequencies of microcontrollers are the highest. The cheapest PCB design, maybe even long interconnect wires.
In such conditions, zero transfer via UART is the last drop to a full barrel of problems.
Yes, I had a project like that, I had to abandon the UART.

[DavidAlfa]

Someone is mistakenly thinking C's null string termination applies to the UART?
Because it doesn't! Zero is just data. Start, "0", stop.
No difference in integrity between sending 0, 1, 0x35, etc, it's the same thing unless you don't know how to properly use it.
Who said cheapest layout, fast MCUs, long wires? Don't start again with fictional facts.

Also, OP evaporated completely.

[langwadt]

Judging by the topic header, we are talking about two microcontrollers in the cheapest possible version - without quartz resonators.
Transmission speed is the highest, frequencies of microcontrollers are the highest. The cheapest PCB design, maybe even long interconnect wires.
In such conditions, zero transfer via UART is the last drop to a full barrel of problems.
Yes, I had a project like that, I had to abandon the UART.

I don't know what you managed to screw up but sending zero with a UART was most definitely not the cause of the problem 

[artag]

Convert the data to a known byte-order at each end and back to the local byte-order at the other end.
There are calls in the C library to do this - you don't need to know what each end does. Note that at one end no work will be done because if the endianness differs one will already be network byte order.
These calls are used in all TCP networking code. You can be sure they will be heavily optimised.
https://stackoverflow.com/questions/36924598/understanding-htonl-and-ntohl

A separate problem is delimiting the records and ensuring data transparency. Typically a procedure called byte-stuffing is used to reserve a few bytes for record management and ensure they will never be sent as data. To do this, it 'escapes' teh reserved bytes by prefixing them with another special byte and sending a modified value.
Finally, use a checksum or CRC to catch any corrupted values.

Some people think this is overkill and use a 'simpler' scheme - often a byte count followed by the bytes of the message. Do not do this. It is foolish. Do it properly, or suffer the 1 byte in 10,000 that gets corrupted and needs the system to be reset before it can recover.

(this is broadly agreeing with nctnico. Converting everything to text is an alternative that may be easier if you don't have much data to send)