Portable Safe Way To Send Struct Between 2 Microcontrollers?

[nctnico]

Convert the data to a known byte-order at each end and back to the local byte-order at the other end.
There are calls in the C library to do this - you don't need to know what each end does. Note that at one end no work will be done because if the endianness differs one will already be network byte order.
These calls are used in all TCP networking code. You can be sure they will be heavily optimised.
https://stackoverflow.com/questions/36924598/understanding-htonl-and-ntohl

These are not suitable as these functions don't work on byte buffers. You'll need similar functions (or macros) which can work with a byte buffer as input or output. The compiler will optimise such functions as far as the platform allows.

I second the suggestion for having checksums over the data.

[SiliconWizard]

Convert the data to a known byte-order at each end and back to the local byte-order at the other end.
There are calls in the C library to do this - you don't need to know what each end does. Note that at one end no work will be done because if the endianness differs one will already be network byte order.
These calls are used in all TCP networking code. You can be sure they will be heavily optimised.
https://stackoverflow.com/questions/36924598/understanding-htonl-and-ntohl

Uh. This is not the standard C library, it's POSIX stuff. POSIX is not the standard C library.
And the probability you have access to POSIX network functions on microcontrollers is close to the absolute zero in general.

The OP's question was "between 2 microcontrollers".
Besides, these functions "convert values between host and network byte order", whatever the "network byte order" means on a given platform supporting POSIX.
I fail to see how that could answer the OP's question here.

Handling byte swap for handling different endianness is pretty basic stuff, it's not rocket science.
The issue is usually not implementation, it's documentation.

[tellurium]

Not sure if that will help the OP, but I guess I'll mention that.
We had a project like OP mentioned - two MCUs, they had to communicate via the UART.
In the end, the solution was - a text-based communication: JSON messages separated by new line characters, with crc32 followed each JSON string. Something like that:

Code: [Select]

{"method":"ble.adv", "params":{....}} 0x12345678 \n
{"id":1, "method":"fs.list"} 0x34567890 \n
{"id":1, "result": ["a.txt", "b.txt"]} 0xaabbccdd \n

I've no clue how big the anticipated data is, what's the expected data exchange rate, etc etc.
The solution above might be feasible for the moderate, or low, data exchange rate. It is human readable, so it is easy to debug/test.

[rhodges]

The UART is designed to work with text. As soon as a zero is encountered, it will signal the built-in end-of-line processing.

NO. I can't use "thumbs down" on your message, so I will approve the others who show you are wrong.

Did you say this before? If so, please stop.

[Fredderic]

As everyone's been saying, byte-swapping is not hard!  Decide on a byte order, document it, and write functions that copy the data between the internal structure and a transmission buffer.  I'd additionally include a structure ID tag and length on the front, plus a CRC on the end as has also been noted (also presumes some kind of error recovery).  It's a little tedious and possibly error prone perhaps, but it's also simple and reliable, and that's what testing is for.  Best of all, it completely removes both endianess and padding concerns since you're directly controlling the wire format.

That seems just basically sensible to me.  Those send and receive functions also allow you to reorder the internal structure fields, adjust padding, even allow for different machine word sizes (as long as you're using something big enough for the data), and allow the compiler to freely do the same, without disrupting the wire format.  It's also quite easy once you have a few, to toss together a little Python to generate those functions out of a JSON schema (they don't need to be pretty), and the schema in turn makes up part of the documentation.

Annoyingly, most of the "standard" formats seem to either go for minimal binary representation, or maximal flexibility at the expense of size, I haven't seen one I'd say is focused on practicality, which is what I'm generally aiming for; knowing how much memory you'll need as soon as possible, not having to parse a structure you don't need to deal with (important in bus situations), and being able to sit and wait until there's enough space in the outbound buffer (or the entire structure has been received) to avoid double-copying or more convoluted processes, etc.

I generally aim to use the endieness of the smallest processor — so it has the least work to do — and to front-load as much of the fixed-size data as possible.  And then for any more flexible fields, I substitute their size into the transmission structure, and attach the variable part(s) to the end (can sometimes omit the size of the last variable part, thanks to having the overall packet size up front).  A small helper function per structure that computes the space it'll take up in the transmission stream also comes in particularly useful; both for prefixing to the data, and making sure there's sufficient space in the transmit buffer.

Personally, I also rather like Protocol Buffers' flexible integer format (comes in rather handy at times — especially for the packet length header field), though they do break the optimisation of having all your fixed data up front.  But it's clean, easy, and more efficient than hex or b64.  (I've also seen UTF-8 encoding hijacked for sending numbers…  it works…  mostly).

Just remember to document all the things, especially if you're eschewing a standard format for something more pragmatic.

[hans]

The UART is designed to work with text. As soon as a zero is encountered, it will signal the built-in end-of-line processing.

NO. I can't use "thumbs down" on your message, so I will approve the others who show you are wrong.

Did you say this before? If so, please stop.

Indeed, it couldn't be farther from the truth. If a zero byte (or any other byte with enough consecutive zeros) is causing framing errors, then that setup is broken. If that is the UARTs fault, then that is someones incompetence.
Obviously its useful to be aware how an UART works, and why e.g. having devices power cycle can cause (recoverable) faults.

Wrt to solution posted by DavidAlfa: I'm not fully into the big endian scheme.. but if I read about older ARM systems having a different byte order for floating point (its not standardized in IEEE754), or different big endian schemes such as be32 and be8 (thus causing bytes to be swapped or not depending on read size)... There can also be memory alignment issues if casting a buffer to some structure pointer, as well. Personally I wouldn't approach it like that anymore.

Its not that hard to write a stream serializer and deserializer. You initialize it with a (circular)buffer and a start index, and then expose readUint8(), readFloat(), etc. functions. You can do all the platform specific implementations in there you ever need. I'm not sure if speed is that much of a concern here, unless one is really trying to save the final cycle on their CPU.
Json, Bson or protobuf is also an option. But it typically drags in a library dependency. For transmission, Json is not that hard.. but reading is usually the more fiddly part. The last time I did that was on Arduino, which had some convenient libraries to handle that. A larger MCU like ESP32 can easily run those functions.

[voltsandjolts]

In addition there will be a lot of complexity with packet synchronization. Yes, your data structure is no longer a structure, but a packet - it has a beginning and an end.

Data structure and packet framing are best treated as separate issues, and thus separate layers of your communication stack, whatever that may be. COBS is an efficient packet framing method which places no restrictions on data content, making for a neat self-contained layer of the stack.

[Siwastaja]

OP went silent, but out of pure interest, I re-ask the question, what is the big-endian microcontroller?

[AVI-crak]

The forward order (from junior to senior) is used on almost all modern architectures. However, the reverse order is used on an impressive number of types of communication interfaces, as well as on a variety of data storage formats.
For example, everything related to data storages uses reverse data order in the description of data structures. Just like the very first punched cards.
A lot of time has passed, a lot of operations are completely automatic at the hardware level. This gives the illusion of external simplicity of data handling.
Until someone goes down to the bottom of the basement.

[hal9001]

Thanks all 

How about specifying the endianness? I guess the compiler will know it's little endian at compile time and make everything for us?

Code: [Select]

struct __attribute__((packed, scalar_storage_order("little-endian")))

This work, many thanks.

struct __attribute__((packed, scalar_storage_order("little-endian")))

I appreciate gcc is the Google of compilers, but there are other games in town and making something dependent on a specific tool isn't really any better than the problem it's solving.

Maybe not a very portable solution still 

Judging by the topic header, we are talking about two microcontrollers in the cheapest possible version - without quartz resonators.
Transmission speed is the highest, frequencies of microcontrollers are the highest. The cheapest PCB design, maybe even long interconnect wires.
In such conditions, zero transfer via UART is the last drop to a full barrel of problems.
Yes, I had a project like that, I had to abandon the UART.

Uart baud rate is 57600. No corruption of uart data, problem is the struct members dont align with the right bytes.

OP went silent, but out of pure interest, I re-ask the question, what is the big-endian microcontroller?

Turns out both micros are both little endian but one is 8 bit the other is 32. The problem is the padding. Im still after a solution that is portable in case I move to another micro thats big endian. Is big endian not so common I shouldnt worry about it?

[Siwastaja]

Turns out both micros are both little endian but one is 8 bit the other is 32. The problem is the padding. Im still after a solution that is portable in case I move to another micro thats big endian. Is big endian not so common I shouldnt worry about it?

While DavidAlfa's endianness attribute would be a little bit "weird" beast, at least packed attribute/pragma is available on any compiler worth using, so even if still not standard C, it's pretty damn common. And despite some claims on this very forum, I have never seen or heard any compiler generating unaligned accesses using a packed struct, but of course being non-standard, it's not guaranteed portable, even if it practically is.

However, if you can freely design the structure, I strongly suggest to make the members naturally-aligned for the 32-bit target (it will obviously be aligned for the 8-bitter, too). Natural alignment generates smallest and fastest code. (Unaligned members need more instructions, which compiler automagically generates for packed structs, or which you manually write using the nctnico's way, with bitshifts etc.)

Natural alignment means that 32-bit members start at 4-byte boundaries, 16-bit members at 2-byte boundaries etc.. You get closest to this (minimum padding required) by simply ordering the members from biggest to smallest (ages old trick). Then, if padding is still needed for natural alignment, you can pad members manually - this has added benefit of it becoming documented, and giving you explicit reserved members you can later start using when extending the protocol. It's also more obvious to zero such explicit padding members (uninitialized compiler-generated padding is a semi-common safety bug). Add a comment to reserved members such as "always write to 0". Then any non-zero value would denote you are using the extended feature.

When you manually pad the struct to natural alignment, compiler has no reason to add more padding, and thus a packed attribute becomes nearly irrelevant (however, see the important point below). It's a good idea to add a few static_asserts() to verify the offsetof() of a few members, and maybe sizeof of your struct. Then any false assumption on your behalf would simply result in an error.

Remember, if you pointer-cast your struct type over a randomly aligned byte buffer, it will only work if the struct type has the packed attribute. If you ever plan doing that, the options are packed structs and restricting yourself to compilers which behave this way (so practically anything sane), or creating a temporary struct object and memcpying to it, or using accessor helper functions which internally use bitshifts, anding and orring.

[DavidAlfa]

How about generating a constant table containing each struct element size?
Then the sending fucntion only need to check this, and swap the bytes or now depednign on the system endianness.
The receiving function would do the same thing.

Code: [Select]

#if !defined(__BYTE_ORDER__) ||  !defined (__ORDER_LITTLE_ENDIAN__) || !defined  (__ORDER_BIG_ENDIAN__)
    #error "__BYTE_ORDER__  not supported by this compiler!"
#endif


struct __attribute__((packed)) typedef struct {
  uint8_t a;
  uint16_t b;
  uint32_t c;
} data_t;

data_t data;

const uint8_t data_sizes[] = {
  sizeof(data.a);
  sizeof(data.b);
  sizeof(data.c);
}

void send_data(void){
  uint8_t * d = (uint8_t *) data;
  for(uint8_t i=0;i<sizeof(data_sizes);i++){
    if(data_sizes[i] == 1){
      Uart_tx(*d); d++;
    }   
   
#if(__BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__)
    else if(data_sizes[i] == 2){
      Uart_tx(*d++);
      Uart_tx(*d++);
    }   
    else if(data_sizes[i] == 4){
      Uart_tx(*d++);
      Uart_tx(*d++);
      Uart_tx(*d++);
      Uart_tx(*d++);
    }
   
#else
    else if(data_sizes[i] == 2){
      Uart_tx(*(d+1));
      Uart_tx(*d);
      d+=2;
    }   
    else if(data_sizes[i] == 4){
      Uart_tx(*(d+3));
      Uart_tx(*(d+2));
      Uart_tx(*(d+1));
      Uart_tx(*d);
      d+=4;
    }
    #endif   
  }
}


[uer166]

Is big endian not so common I shouldnt worry about it?

Yeah pretty much, unless you can name a specific MCU that you know you might use. It would certainly be a special snowflake, I just wouldn't worry about it at all. Thanks for answering the endianness question because big endian is just not common.

[NorthGuy]

This is not how it works. You do not start from writing a C code. You start from designing a protocol. UART transfers byte streams. You need a method to embed structures (packets) in these stream, perhaps check integrity (or not), perhaps use packet numbering (or not). You need to find a way to find these structures in the UART stream on the receiving end. You design a protocol which can do all of that.

Only then you write the C code which implements the protocol on one MCU (you will need two instances of it of course) and you test it, perhaps revise your protocol in process. Now you have a working communication.

If your goal is to port it to other MCU(s), you try to make it portable from the very beginning. You design your protocol so that you wouldn't have alignment problems, wouldn't depend on idiosyncrasies of your particular MCU, would be easy to compose and parse. You write your C code accordingly. Then, by the time you have created a working model with two MCUs, your code is already portable. When you need to port to a different MCU, you simply use the same code. You probably need to replace functions which send/receive bytes, and you're done. Porting was easy because you thought of that from the very beginning, when you started designing your protocol.

[pqass]

A separate problem is delimiting the records and ensuring data transparency. Typically a procedure called byte-stuffing is used to reserve a few bytes for record management and ensure they will never be sent as data. To do this, it 'escapes' teh reserved bytes by prefixing them with another special byte and sending a modified value.
Finally, use a checksum or CRC to catch any corrupted values.

Some people think this is overkill and use a 'simpler' scheme - often a byte count followed by the bytes of the message. Do not do this. It is foolish. Do it properly, or suffer the 1 byte in 10,000 that gets corrupted and needs the system to be reset before it can recover.

Fortunately with UARTs you can send a BREAK as an end-of-message indicator so byte-stuffing won't be necessary.

If your transmitter doesn't directly support it, you can just lower the baud rate temporarily and send a zero byte.  The receiver will see that as a zero byte with a framing error, that is, the stop bit wasn't detected when expected.

The sender could send a binary or human-readable message consisting of the following fields: 
           magic number, CRC, message sequence, type, version, marshaled structure, BREAK

The receiver can loop reading all characters into a buffer^*.  Only after a BREAK is received will it validate the buffer against the sent CRC and if they match, the buffer can be processed.  Another message can be sent back to report success or error.

Any noise before the magic number can be ignored. If the magic number is found but CRC doesn't match then a error message can be returned, otherwise, the buffer is discarded without reply. You don't want to generate a reply storm on a noisy line.  The sender would just re-send if it didn't get any reply.

^* circular buffer holding the last n characters where n > largest message.

[PlainName]

This is not how it works. You do not start from writing a C code. You start from designing a protocol. UART transfers byte streams. You need a method to embed structures (packets) in these stream, perhaps check integrity (or not), perhaps use packet numbering (or not). You need to find a way to find these structures in the UART stream on the receiving end. You design a protocol which can do all of that.

Only then you write the C code which implements the protocol on one MCU (you will need two instances of it of course) and you test it, perhaps revise your protocol in process. Now you have a working communication.

If your goal is to port it to other MCU(s), you try to make it portable from the very beginning. You design your protocol so that you wouldn't have alignment problems, wouldn't depend on idiosyncrasies of your particular MCU, would be easy to compose and parse. You write your C code accordingly. Then, by the time you have created a working model with two MCUs, your code is already portable. When you need to port to a different MCU, you simply use the same code. You probably need to replace functions which send/receive bytes, and you're done. Porting was easy because you thought of that from the very beginning, when you started designing your protocol.

I don't usually quote whole posts, but this is worth repeating because it's The Way To Do It.

[tggzzz]

This is not how it works. You do not start from writing a C code. You start from designing a protocol. UART transfers byte streams. You need a method to embed structures (packets) in these stream, perhaps check integrity (or not), perhaps use packet numbering (or not). You need to find a way to find these structures in the UART stream on the receiving end. You design a protocol which can do all of that.

Only then you write the C code which implements the protocol on one MCU (you will need two instances of it of course) and you test it, perhaps revise your protocol in process. Now you have a working communication.

If your goal is to port it to other MCU(s), you try to make it portable from the very beginning. You design your protocol so that you wouldn't have alignment problems, wouldn't depend on idiosyncrasies of your particular MCU, would be easy to compose and parse. You write your C code accordingly. Then, by the time you have created a working model with two MCUs, your code is already portable. When you need to port to a different MCU, you simply use the same code. You probably need to replace functions which send/receive bytes, and you're done. Porting was easy because you thought of that from the very beginning, when you started designing your protocol.

I don't usually quote whole posts, but this is worth repeating because it's The Way To Do It.

The OP hasn't addressed the issue of why he wants to send one hardware/compller/flags dependent set of bits to a processor with different hardware/compller/flags. That's philosophically and practically wrong-headed.

Sending the information represented by the set of bits is perfectly sensible, of course.

[Siwastaja]

The OP hasn't addressed the issue of why he wants to send one hardware/compller/flags dependent set of bits to a processor with different hardware/compller/flags. That's philosophically and practically wrong-headed.

Sending the information represented by the set of bits is perfectly sensible, of course.

We can guess: OP has specified one of the controllers is a 8-bitter. These are usually very constrained in resources (CPU time, program code memory and RAM), so using a format which is efficient for this 8-bit CPU and requires minimum amount of parsing there (and possibly more parsing at the 32-bit side) seems sane. For an 8-bitter, JSON, for example, is probably out of question.

[peter-h]

To the OP: just snprintf the values to a buffer (buffers starts with "*") send them out, append a special char e.g. "$" to the buffer, and at the other end discard data until you get a "*", then receive until you get a "$", obviously with buffer overflow detection etc, and then sscanf them back in.

All those complicated schemes...

If the CPU is a Z80 running at 4MHz, and no float code, scale the data appropriately and use integers. 32 bit ints are extremely efficient to output and to input.

[fchk]

There is a lightweight way of transferring data between diifferent architectures since 1995: XDR - the External Data Representation. Used in thing like NFS, RPC, ZFS, ...
See https://en.wikipedia.org/wiki/External_Data_Representation
Standard: https://datatracker.ietf.org/doc/html/rfc4506

[tggzzz]

To the OP: just snprintf the values to a buffer (buffers starts with "*") send them out, append a special char e.g. "$" to the buffer, and at the other end discard data until you get a "*", then receive until you get a "$", obviously with buffer overflow detection etc, and then sscanf them back in.

All those complicated schemes...

If the CPU is a Z80 running at 4MHz, and no float code, scale the data appropriately and use integers. 32 bit ints are extremely efficient to output and to input.

To avoid having printf() code in the 8-bit MCU, send each byte as two hex characters. No rocket science required.

[nctnico]

To the OP: just snprintf the values to a buffer (buffers starts with "*") send them out, append a special char e.g. "$" to the buffer, and at the other end discard data until you get a "*", then receive until you get a "$", obviously with buffer overflow detection etc, and then sscanf them back in.

All those complicated schemes...

If the CPU is a Z80 running at 4MHz, and no float code, scale the data appropriately and use integers. 32 bit ints are extremely efficient to output and to input.

To avoid having printf() code in the 8-bit MCU, send each byte as two hex characters. No rocket science required.

Don't avoid snprintf but use a minimalistic one. Which is likely already provided as part of the standard libraries in various levels of functionality versus size. Re-inventing the wheel (which often leads to unmaintainable code) is not necessary.

[Siwastaja]

To avoid having printf() code in the 8-bit MCU, send each byte as two hex characters. No rocket science required.

This has added benefit of being a constant size conversion i.e., it exactly doubles the size of binary data so buffer sizes and offsets to buffers are easy to calculate. You can then use rest of the characters (e.g. space, comma) as delimiters (message start, message end, whatever is needed). Or, for example, whitespace can be ignored by parser and used as human aid.

Indeed, simple raw hexadecimal tends to be the sweet spot between machine and human readability/writability in very resource constrained systems. It might not be fun to look at hexadecimal output at terminal screen, but it sure is possible. Binary format tends to need writing of some helper tools, after which it is both efficient and easy to use, but some more upfront work.

[peter-h]

To avoid having printf() code in the 8-bit MCU, send each byte as two hex characters. No rocket science required.

Even if you use floating point, you don't need to use printf/scanf provided you settle on a format like 1234.567 and with fixed # of digits.

This is trivial to output and equally trivial to receive.

You treat each half as an int and do itoa for output (two of those, with a '.' between) and atoi (again 2x) for input.

Many years ago I was doing an HPGL parser on an 8MHz Z180 on which doing this was literally 1000x faster than scanf (on an IAR C compiler). It is also very easy in assembler but no point unless the micro is something like a 90S1200 but the if it was that, you aren't likely to use C anyway. I had to do this because the data was arriving at 115k baud and the IAR sscanf was about 100ms!

[tggzzz]

To the OP: just snprintf the values to a buffer (buffers starts with "*") send them out, append a special char e.g. "$" to the buffer, and at the other end discard data until you get a "*", then receive until you get a "$", obviously with buffer overflow detection etc, and then sscanf them back in.

All those complicated schemes...

If the CPU is a Z80 running at 4MHz, and no float code, scale the data appropriately and use integers. 32 bit ints are extremely efficient to output and to input.

To avoid having printf() code in the 8-bit MCU, send each byte as two hex characters. No rocket science required.

Don't avoid snprintf but use a minimalistic one. Which is likely already provided as part of the standard libraries in various levels of functionality versus size. Re-inventing the wheel (which often leads to unmaintainable code) is not necessary.

Neutered printf() is an option, but hex chars are very simple/small/fast and may be sufficient. Who knows