How to prevent the compiler messing with a structures allignment

[madires]

Why we don't use structs as protocol datagrams:
- padding
- alignment
- endianness

Instead we put everything into a datagram buffer or extract everything from a received datagram, using functions like htonl(), htons(), ntohl() and ntohs() for example.

[ataradov]

You have reasonable and predicable control over the padding and alignment. Endianness is a bit harder, but also solvable in a much cleaner way.

I don't understand this fear of packing the structures. It results in a much cleaner and easier to maintain code where you have exact description of the data in a language format instead of extracting bytes from a buffer and potentially messing up the offsets when the data format changes.

[PlainName]

... the interrupt handler will update the variable, if it is in the middle of boing read then that may not be nice. but if it is all 4 byte aligned then the read/write will be one operation so no entrails.

IME this will lead to tears, usually after you've forgotten why it has to be aligned and using a different system where that doesn't apply anyway, so the problem isn't immediately obvious. It takes a while to figure out what's happening and then a longer while to figure out why it's broken now and wasn't before.

[madires]

I don't understand this fear of packing the structures. It results in a much cleaner and easier to maintain code where you have exact description of the data in a language format instead of extracting bytes from a buffer and potentially messing up the offsets when the data format changes.

One reason is portability between CPU/MCU architectures. Compilers can cause some fun too (other compiler, other version). Also, protocols have their own idea on how data is arranged which might be very different from the CPU/MCU architecture you're writing code for.

[Simon]

... the interrupt handler will update the variable, if it is in the middle of boing read then that may not be nice. but if it is all 4 byte aligned then the read/write will be one operation so no entrails.

IME this will lead to tears, usually after you've forgotten why it has to be aligned and using a different system where that doesn't apply anyway, so the problem isn't immediately obvious. It takes a while to figure out what's happening and then a longer while to figure out why it's broken now and wasn't before.

Well if you can explain how to "mutex" in a simple C environment. When you think about it it is actually a lot more complicated than "just set a flag to grant/deny access to variable" enter OSes and ya ya ya I'll just use a single board computer next for what it was not intended for.

[nctnico]

All I need is that it be repeatable. If people are not competent then they should not attempt the job. We originally set out to be 4 byte aligned, this also aids in the variables being updated as another "process" in the interrupt handler will update the variable, if it is in the middle of boing read then that may not be nice. but if it is all 4 byte aligned then the read/write will be one operation so no entrails.

The problem is that when using packed structs, this atomic operation is not guaranteed. I have seen compilers emit code that reads & shifts a multi-byte struct member from a buffer one byte at a time which is not an atomic operation as you expect.

[ejeffrey]

It really depends on how you use it.  If you use the packed structure only for serialization, its fine.  If you use it for anything else, it can start to be a problem: pointers to members are a different data type than pointers to the native types, in general, packed structures are also unaligned which means potentially every access requires byte-wise manipulation by the compiler.  As mentioned, if you use the structure from an ISR you can't access it atomically.

One way to solve this is to have two versions of the struct, a packed and unpacked, and only use the packed one for serialization.  You then do an element-wise copy from one to the other when serializing.  It's probably the best way to handle this if you need to match a specific layout, and it's easy enough to add endian conversion at this point if needed.  It also reliably does exactly the same as the manual byte shuffling code people advocate writing by hand, but possibly more efficiently.

But I generally wouldn't do that unless I needed to match a pre-existing data type.  If I am working on PCs exclusively, I would just use a purpose built serialization format: probobuf, cap'n proto, or json.  On a microcontroller I typically just serialize native (non-packed) structs.  These days, alignment rules are standard enough that if you use stdint fixed width types, everything will get the same alignment on every platform I care about.  You can add explicit padding members for clarity or if you want it to be compatible with 8- or 16- bit ABIs that don't use natural alignment.  This is certainly not standards guaranteed behavior, but it works well enough for me.

The advantage here is that you get fast access and avoid any other issues with packed data types.

If I only need microcontroller->PC, generating JSON (or YAML) actually isn't too bad, but I wouldn't want to parse JSON on a microcontroller. 

[PlainName]

Well if you can explain how to "mutex" in a simple C environment.

Depends on what you're doing but it could be as simple as reading until you get two values the same. More 'proper' would be to use some variable with atomic access to act as the mutex - this would be in your BSP so is dependent on, and likely changes with, your hardware but the interface to it would be the same for all. Move to other hardware and all that should change is the BSP (which is why it's so named).

[SiliconWizard]

All I need is that it be repeatable. If people are not competent then they should not attempt the job. We originally set out to be 4 byte aligned, this also aids in the variables being updated as another "process" in the interrupt handler will update the variable, if it is in the middle of boing read then that may not be nice. but if it is all 4 byte aligned then the read/write will be one operation so no entrails.

The problem is that when using packed structs, this atomic operation is not guaranteed. I have seen compilers emit code that reads & shifts a multi-byte struct member from a buffer one byte at a time which is not an atomic operation as you expect.

Note that if you can use C11, you can use stdatomic - define the struct members that require atomic operation as an stdatomic data type and use the appropriate functions for access.
Of course you still have to be aware of the added cost, if it matters.

[DavidAlfa]

Use packed struct and then send using uint8_t* pointer with data size to avoid endian issues.

Now I had a doubt. Will the compiler handle copying between same structures, one packed and other not?

Code: [Select]

typedef struct { ... } data_t;
__attribute__((packed)) data_t data_packed;
data_t data_unpacked;

Would this work properly?

Code: [Select]

data_unpacked = data_packed;
data_packed = data_unpacked;

I don't think so! Never tried, I guess the compiler will spit some nasty warnings out.

[Shonky]

I'd expect the compile to barf on that straight away as data_packed and data_unpacked are essentially not that same type.

[ataradov]

You can't set packed attribute to individual variables, it is a type attribute. GCC will ignore it and issue a warning that it was ignored.

But generally, if you do a lot of processing on the variables, it may make sense to have two versions and use one for the storage and one for the transmission. You would still have to assign all the members manually, but in the end it is still cleaner code than building a buffer by hand. It is also more efficient code is some members end up being aligned by chance in the packed structure. Plus is is the best place to add endian swap in a clean way if needed.

[djacobow]

Why do these types of questions always end up like this: a thread with people insulting each other.

Here's the thing: lots of people through history have needed to move data over comms channels. It is, to a first approximation, "easy" ... and then invariably this gets effed up one way or another.

If you are a clever, or just think you are, then by all means, just send packed structures, or do your own byte order swapping, or whatever you need for your source, channel, and destination, and be on your merry way.

Protobufs were invented by smart people at early Google who found that despite being smart, the effed this up regularly. So they made a protocol to make it easy to serialize and deserialize complex data structures. Furthermore they made a tool to make it easy to access the structures in whatever language they were using on whichever end. There are other tools for this purpose out there, but you should ask yourself why do they exist at all if it's so simple.

The next level up of ways to transfer data is to stringify it. When you have a lot of structured data to send, and the structs vary for example lists change length, and dictionaries have more keys, then json or xml or even csv can be appealing. And it can be much easier to debug. But no doubt it has a lot of overhead regarding the size of the data transmitted.

Original point was only that there are options, and if your problem really is simple, and if you actually know what you're doing, then of course you can send packed structs. And, the easiest way to deal with endianess is to have a system where all the endpoints are the same endianess. Then you can do nothing! And in fact people get away with that for a long time, until some new device shows up and is different, and then their code breaks. This has happened before, and it will happen again. And again, and again, and again, forever.

[nctnico]

You would still have to assign all the members manually, but in the end it is still cleaner code than building a buffer by hand.

It is easy to create a few small functions that put a certain type at the end of a buffer and increment the buffer pointer. That way adding / deleting a member just works out of the box.

something like put_int32(value, **buffer_pointer)
It byte-shifts 'value' into 4 bytes of the buffer and increments the buffer pointer accordingly. Unless the formatting of the buffer needs to adhere to a specification, you don't have to care where each element is at.

And the compiler being able to do aligned accesses on a buffer is far less likely to happen. How can the compiler know a buffer pointed to by a void pointer or byte type pointer is aligned during runtime? And how much do you actually gain from a single aligned access versus loading 4 bytes when processing a message every once in a while?

[Siwastaja]

Simon, ignore everybody else except ataradov and just simply use

Code: [Select]

struct __attribute__((packed)) type_name {
...
};

I will guarantee it works and this solves your problems without overthinking nonexistent issues. This also minimizes the risk of human errors, and dependencies on external work. Basically, you hand out all the low-level details like alignment etc. to the compiler, and pay a performance penalty for this. Premature optimization is evil; don't do it; instead, describe the data type as a data type and let the compiler handle the access.

I have been doing exactly this in dozens and dozens of different projects, never any problem for the last decade. None. Zero. Believe me and ignore the naysayers.

(This will be my only post in this thread. Either you get it, or you don't.)

[JPortici]

Simon, ignore everybody else except ataradov and just simply use

Code: [Select]

struct __attribute__((packed)) type_name {
...
};

I will guarantee it works and this solves your problems without overthinking nonexistent issues. This also minimizes the risk of human errors, and dependencies on external work. Basically, you hand out all the low-level details like alignment etc. to the compiler, and pay a performance penalty for this. Premature optimization is evil; don't do it; instead, describe the data type as a data type and let the compiler handle the access.

I have been doing exactly this in dozens and dozens of different projects, never any problem for the last decade. None. Zero. Believe me and ignore the naysayers.

(This will be my only post in this thread. Either you get it, or you don't.)

+1
And i just want to add that if one of the entities receiving/sending the data is a windows PC with the software compiled with GCC/MinGW, before the definitions of the types you have to add
#pragma pack(1)
then after the definition of the types you have to add
#pragma pack()
or MinGW will happily ignore the packed attribute

[madires]

Why do these types of questions always end up like this: a thread with people insulting each other.

youngster: I use this shortcut. Works great!
greybeard: Some words of wisdom ...
youngster: Meh. It works and it's simpler.
greybeard: Sooner or later you'll find out the hard way.
youngster: I don't need your advise!
greybeard: Ungrateful youngster!

A never ending story.

[AVI-crak]

I don’t understand how you can discuss an abstract problem blindfolded ...

[DavidAlfa]

Greybeard = wisdom? Sometimes, but very often it's just fixation/obsession/habits, never-happenning 1 in 10 million things, and "I always did that way " rule,  as you age it gets worse and worse and some won't accept any other method except their's.

Probably I'm getting roasted again, but I don't care!

[gf]

Simon, ignore everybody else except ataradov and just simply use

Code: [Select]

struct __attribute__((packed)) type_name {
...
};

I will guarantee it works and this solves your problems without overthinking nonexistent issues. This also minimizes the risk of human errors, and dependencies on external work. Basically, you hand out all the low-level details like alignment etc. to the compiler, and pay a performance penalty for this. Premature optimization is evil; don't do it; instead, describe the data type as a data type and let the compiler handle the access.

And in C++ with the Boost.Endian library you can even go further and write e.g.

  #include <boost/endian/arithmetic.hpp>
  using namespace boost::endian;

  struct S {
    big_uint16_t a;
    little_int32_t b;
  };

which leads to a packed struct as well, and which does the proper endian conversion on the fly when you access the members.

[Karel]

C++ is a horrible language. Way too complex and you don't know what it's doing under the hood...
Thanks but I'll stick with C.

[peter-h]

Data intended to go over a serial channel should always be assembled and sent byte-serially.

Compose a buffer (defined as bytes e.g. uint8_t fred[100]; ) and load it with your data.

At this point you will need to think about which of say 4 bytes of a uint32_t goes out first. So e.g. if you have

uint32_t joe;

you might do

fred[0] = joe & 0xff;    // loads least significant byte of joe into 1st byte of buffer
fred[1] = (joe>>8 ) & 0xff;
fred[2] = (joe>>16) & 0xff;
fred[3] = (joe>>24) & 0xff;  // loads most significant byte of joe into 4th byte of buffer

Probably the "&0xff" is not required since the array fred is defined as bytes, but it makes it clear, and covers the "integer promotion" thingy in C.

The >>24 shifts are just 1 clock cycle because an arm32 has a 32 bit barrel shifter.

The other advantage of preparing a buffer is that you can breakpoint the code and examine it before it gets sent, and similarly at the other end.

A lot of people love structures, typedef structures, structures of structures, and before long nobody else can follow what is going on.

In your case you are probably stuck with some predefined protocol format definition, but an even more portable way would be to generate a decimal string, with sprintf ("%d etc. Or ltoa() etc. Then pick it up at the far end with atoi() or atol().

[abeyer]

Note that if you can use C11, you can use stdatomic - define the struct members that require atomic operation as an stdatomic data type and use the appropriate functions for access.
Of course you still have to be aware of the added cost, if it matters.

I'm curious, do implementations of stdatomic deal with unaligned access in practice? They're certainly not guaranteed to

[cv007]

Note that if you can use C11, you can use stdatomic

An M0 (assuming a SAMC in use here) cannot make use of the atomic headers, in addition I believe you cannot use packed structs with atomic as it will assume all atomic vars are aligned. The compiler will emit instructions to get atomic and 'forget' what it knows about the packed struct (for example emits a ldrh even when packed struct alignment requires 2 ldrb's).

The atomic headers for c/c++ are pretty much worthless for the mcu when you have an M0 in the mix of the mcu's you use, and want a universal solution. I assume that's why you seldom see any manufacturer's libraries using anything other than the 'old fashioned' interrupt protection.

Simple example of interrupt protecting any scope, which in c++ ends up being nicer than the typical macros used in C-
https://godbolt.org/z/WMG7MjvTY
Not sure how good the example data structs are, but was just created on the fly to make the example a little more relevant.

[Veteran68]

C++ is a horrible language. Way too complex and you don't know what it's doing under the hood...
Thanks but I'll stick with C.

I disagree, but it's going to be a personal thing. The cult of programming language is one of the strongest forces known to man.

A little career history. I was an assembler programmer before I was a C programmer, and looked down my young and arrogant nose at those "high level" C programmers who couldn't code in a "real language." Then I had to learn C (because you know, job) and quickly learned to appreciate the benefits it provided over assembler (even though I still have a soft spot for ASM and it still has it's place). For years I was an embedded systems engineer who wrote firmware code in C, ran it through the compiler to get an ASM dump, then walked through and hand-optimized the ASM to squeeze out every clock cycle I could before running it through the assembler to get a binary.

I saw the C++ craze taking off, and once again, I looked down my nose saying "everything you do in C++ I can do in C." Then when I had to learn C++, the initial learning curve to "get it" was huge. I struggled. Then one day, the OOP light bulb went off and I was sold. I still wrote C from time to time for lighter stuff, and generally did prefer it for really tight code requirements, but as I wrote heavier, general/business computer software I appreciated the value of OOP and C++.

Years later, when the Java craze took off, I once again frowned at it. However, due to a big project that was the first Java project in the company (we're predominantly Java these days), I had to learn it. I really fell in love with "write once, run anywhere" which I was skeptical of at first but quickly learned was 99% true. I was very familiar with writing C/C++ code that attempted to target multiple OSes and platforms (x86, 68K, 6502, 6809, and then DOS, Windows, Unix/AIX, Linux, Mac, IBM mainframe OS/390) and all the hassles that entailed no matter how much you stuck to "standard" C. Whereas with Java, I could compile one time and carry the bytecode around to any of those business platforms we had and run it as-is. Sure, Java has its drawbacks, mainly in startup time, but well-written Java once started up was quite performant. I had a demo codebase I used to demonstrate C++ performance and Java performance with the same algorithm, and with certain workloads, due to JVM optimizations, some Java code would actually meet or exceed C++ performance.

Point being, I finally learned after several iterations of this life experience, that languages evolve and all bring their advantages (and disadvantages) depending on the use-case. For the past decade+ I've been on the management side and mainly write code these days only for my own benefit, but still consider myself a polyglot programmer who embraces new languages. I've always said that any software engineer worth his salt should be EXCITED to try new languages, and not resist them.

I get that most on this forum are hardware guys that tolerate software as a necessary evil, so I get why you may not share my love for multiple languages. But I've always been surprised just how many in the software field (including many greybeards as previously mentioned) have the same attitude about new/different languages.