Why is loading a multi byte scalar variable from some address so complicated?

[peter-h]

This seems to be the standard construct

uint32_t fred;
uint8_t buf[512];
fred = *(volatile uint32_t*) &buf[24];

The 32 bit integer fred is stored in buf[24] to buf[27];

I know this is endian-dependent but let's forget that for now. If you wanted endian-proof code you would do something like

Code: [Select]

fred = buf[24]|(buf[25]<<8)|(buf[26]<<16)|(buf[27]<<24);
which runs fast enough on a CPU with a barrel shifter.

I've been programming in C for 2-3 years now and have written a lot of code, which works 100%, but I avoid stuff I can't understand, and in this case I don't get the thinking behind why

Code: [Select]

*(volatile uint32_t*)
is needed.

I use that construct all over the place e.g. flash programming in a boot loader, RAM tests, etc. Obviously it works.

I suppose one could have also used

Code: [Select]

memcpy (&fred, &buf[24],4);
 

[Nominal Animal]

If you wanted endian-proof code you would do something like

Code: [Select]

fred = buf[24]|(buf[25]<<8)|(buf[26]<<16)|(buf[27]<<24);
which runs fast enough on a CPU with a barrel shifter.

Wrong conclusion.  Your C compiler will take that expression, and optimize it to a single load or a single load with byte swaps if possible (if it can verify the alignment is sufficient for 32-bit loads), and only when it cannot, will it compile to single-byte loads.

The above pattern is ubiquitous, and therefore well recognized by both GCC and Clang on all architectures where it can be simplified to an unaligned 32-bit load.

An alternate option is to use two aligned 32-bit loads, and rotate the target 32-bit word into the upper or lower one.  This is rarer, and not easily optimized by current C and C++ compilers to optimal code, so this approach is usually only done for bit streams:

Code: [Select]

// Little-endian byte order and bit indexing, 32-bit architecture
uint32_t  get_u32c(const void *buf, uint32_t  bit_offset)
{
    const uint32_t  w0 = *(uint32_t *)(((uintptr_t)buf + bit_offset/8) & (~(uintptr_t)3)),
                    w1 = *(uint32_t *)(((uintptr_t)buf + bit_offset/8 + 4) & (~(uintptr_t)3));
    const uint_fast8_t  shift = (8 * (uintptr_t)buf + bit_offset) & 31;
    return (w0 >> shift) | (w1 << (32 - shift));
}

// Little-endian byte order and bit indexing, 64-bit architecture
uint32_t  get_u32d(const void *buf, uint32_t  bit_offset)
{
    const uint64_t  w = *(const uint64_t *)(((uintptr_t)buf + bit_offset/8) & (~(uintptr_t)7));
    return (uint32_t)(w >> ((8*(uintptr_t)buf + bit_offset) & 63));
}

A third option is to copy the data to an aligned unit:

Code: [Select]

uint32_t  get_u32b(const void *buf)
{
    union {
        uint32_t  u32;
        unsigned char  c[4];
    } result = { .c = { ((const unsigned char *)buf)[0],
                        ((const unsigned char *)buf)[1],
                        ((const unsigned char *)buf)[2],
                        ((const unsigned char *)buf)[3] } };
    return result.u32;
}
While this is not as common as the or-of-shifted-bytes one, GCC and Clang do optimize the last one to a single load on x86-64 (which does allow unaligned 32-bit loads).

You can explore all four versions here at Compiler Explorer (includes the source of above).
You don't really need to be very assembler-savvy; comparing the number of instructions in the different variants on your preferred architecture and compiler is sufficient.

I don't get the thinking behind why

Code: [Select]

*(volatile uint32_t*)is needed.

It tells the compiler that it is not allowed to deduce from surrounding code what the data in the buffer is, or ought to be.

It is necessary when the data in the buffer might be modified by something that the compiler does not see/cannot deduce is happening, for example an interrupt, DMA transfer, or hardware state changes.

[peter-h]

Interesting - thanks.

I did know what "volatile" does but I have seen that construct used all over the place where there is no realistic/meaningful possibility of the source memory being modified by another process. For example ST use it for CPU FLASH programming. But I guess there may be a reason there because one is writing to a memory address which can change after being programmed!

So

Code: [Select]

fred = *(uint32_t*) &buf[24];would have done the same job.

It sounds like

Code: [Select]

memcpy (&fred, &buf[24],4);would also get optimised by the compiler into some simple code.

I've done decades of assembler but never really done it on the arm32.

[magic]

This seems to be the standard construct

I doubt it is even standard compliant, unless uint8_t happens to be equivalent to char, and it doesn't need to be.

[rstofer]

I did know what "volatile" does but I have seen that construct used all over the place where there is no realistic/meaningful possibility of the source memory being modified by another process. For example ST use it for CPU FLASH programming. But I guess there may be a reason there because one is writing to a memory address which can change after being programmed!

I thought the 'volatile' part of the declaration prevented the compiler from optimizing away writes to memory/registers without subsequent reads.

[langwadt]

Interesting - thanks.

I did know what "volatile" does but I have seen that construct used all over the place where there is no realistic/meaningful possibility of the source memory being modified by another process. For example ST use it for CPU FLASH programming. But I guess there may be a reason there because one is writing to a memory address which can change after being programmed!

volatile also tells the compiler to do the write even when it thinks it isn't needed

[DavidAlfa]

But how are ensuring buffer[24] is 32-bit aligned?
Unless you tell the compiler to align buf, it could be placed in any way, casting a byte address as int32 will potentially cause a misaligned access, triggering an exception depending on the system.
The only safe way I can think off would be:

__attribute__((aligned(4))) uint8_t buf[512];

That way you could cast buffer[0, 4, 8, 12, 16, 24...] as int32 safely.

[Nominal Animal]

unless uint8_t happens to be equivalent to char, and it doesn't need to be.

It just happens to be equivalent to unsigned char on all architectures that GCC, Clang, et al. currently support.  Which is the reason it is so often used as equivalent to unsigned char, the char type having special provisions regarding storage representation in the C standard.

I did know what "volatile" does but I have seen that construct used all over the place where there is no realistic/meaningful possibility of the source memory being modified by another process.

Well, it is possible for the contents of the Flash memory to be changed in an interrupt context, or even due to a hardware fault or a cosmic ray (which is just radiation not absorbed by the atmosphere).

If code wants to be bulletproof, i.e. shield even against the unrealistic/nonsensical situation where the contents indeed are modified concurrently, it uses the (*(volatile type*)pointer) idiom.

Just because you and I do not see a realistic/meaningful possibility of the memory being modified in unseen ways, does not mean there is none such. 

Besides, it costs nothing.  It is like the way I like to use static inline for accessor functions, and static for local functions, even though I know perfectly well the two are exactly equivalent: static inline is exactly as likely to be inlined or not as plain static is.  In some cases, it is used as a "cognitive load reducer" for those reading the code.

I can imagine being a vendor library developer, and sprinkling volatile even in places where not strictly required (because the compiler will have no ancillary knowledge to avoid the access anyway), just to avoid questions like "why isn't there a volatile here? I think it is causing a bug in my code" because they do not understand the true meaning of the volatile keyword.

I thought the 'volatile' part of the declaration prevented the compiler from optimizing away writes to memory/registers without subsequent reads.

Technically,

Accesses to volatile objects are evaluated strictly according to the rules of the abstract machine.

In for example C11 footnote 111, regarding assignments, the standard says "The implementation is permitted to read the object to determine the value but is not required to, even when the object has volatile-qualified type."

So, thinking that volatile forces the compiler to do the access is not actually correct; it only forces the compiler to generate code that behaves strictly according to the rules of the C standard abstract machine.

It is only on current hardware architectures, including all architectures supported by GCC, Clang, Intel Compiler Collection, etc., that the only meaningful way to ensure the rules of the C standard abstract machine are followed, is to ensure that
    *(volatile *type)pointer;
and
    expression = *(volatile *type)pointer;
generates code that explicitly loads a value of type type from pointer, and that
    *(volatile *type)pointer = expression;
generates code that explicitly stores the value of the expression at pointer.

Neither is guaranteed by the C standard.  Both are just practical results on how the C abstract machine can be implemented in current architectures.
If one had followed e.g. LKML in the last decade or two, one would remember several threads on how speculative execution and compiler optimizations affect this.  Currently, there are several details where all the above compilers agree and produce effectively the same code, even though the C standard says (or can be interpreted as saying that) the Behaviour is Undefined; just because the compiler users managed to convince the compiler developers that there was a single sane useful use case, and that practical use case overrides any committee-sitters opinions.

This is exactly why I do value the C standard, but believe the practical reality overrides the theory outlined by the standard.  (Indeed, in the past, up to C99, the standard only codified existing behaviour agreed upon by multiple compilers; it was only in C11 that we got Annex K and "new stuff" not implemented by any compiler, because of commercial interests by a single company.)

I do not mean to imply in any way that understanding the C standard would not be useful, because it definitely is, and I believe is quite important for anyone writing any kind of portable code, or code compiled with anything except a specific version of a specific compiler.  I just do not think it is the last word on anything: the last word is the actual tools we use, the practical real world.  I call those who do believe the standard is or should be the last word language-lawyers, and it is an unfair and derisive term, but as I see it, C is and has to be a practical tool and not a theoretical one, because it is still the closest thing we have to a good embedded and systems programming language.

[Nominal Animal]

But how are ensuring buffer[24] is 32-bit aligned?
Unless you tell the compiler to align buf, it could be placed in any way, casting a byte address as int32 will potentially cause a misaligned access, triggering an exception depending on the system.
The only safe way I can think off would be:

__attribute__((aligned(4))) uint8_t buf[512];

That way you could cast buffer[0, 4, 8, 12, 16, 24...] as int32 safely.

I prefer making buffers with alignment requirements the native type, i.e. uint32_t buf[512/4]; here.  (That is, because the data is accessed as uint32_t in a key point, not because I assume uint32_t is 32-bit aligned; it may not be.)

To access any byte/char in the buffer, one can always use ((unsigned char *)buf)[index] (to access a value) or ((unsigned char *)buf + byte_offset) (to get a pointer to a specific byte/char).  So, it is not a limitation at all, but does affect how one intuitively thinks about the buffer.

The reason for my preference is that it seems that compilers can more efficiently optimize access to the array members this way.  That is, they seem to generate better code, at least for the use cases I've tried.

Again, Compiler Explorer aka godbolt.org is an excellent resource for this, because it lets one explore the exact machine code different versions and different compilers generate for specific architectures with specific compiler options.

[newbrain]

unless uint8_t happens to be equivalent to char, and it doesn't need to be.

It just happens to be equivalent to unsigned char on all architectures that GCC, Clang, et al. currently support.  Which is the reason it is so often used as equivalent to unsigned char, the char type having special provisions regarding storage representation in the C standard.

I know your stance about the standard, but as an answer to magic, the standard is what actually mandates the equivalence between uint8_t and unsigned chars.
It's not that it "just happens" to be equivalent...

• If uint8_t exists, it is exactly 8 bit wide (no padding allowed). "7.20.1.1 Exact-width integer types"
• 8 is also the minimum number of bits for any object that is not a bit field, namely CHAR_BIT. "5.2.4.2.1 Sizes of integer types <limits.h>"
• CHAR_BIT is also by definition the bit width of a char. "6.2.6 Representations of types"
• So, (unsigned) char cannot be larger than an uint8_t (as it would violate 2.) and cannot be smaller (as the minimum for CHAR_BIT is 8 ).
• Hence, if  uint8_t exists, it must be equivalent to unsigned char

[newbrain]

Technically,

Accesses to volatile objects are evaluated strictly according to the rules of the abstract machine.

In for example C11 footnote 111, regarding assignments, the standard says "The implementation is permitted to read the object to determine the value but is not required to, even when the object has volatile-qualified type."

So, thinking that volatile forces the compiler to do the access is not actually correct; it only forces the compiler to generate code that behaves strictly according to the rules of the C standard abstract machine.

I beg to differ here.
Footnote 111) needs to be read in the very specific context it is placed:

An assignment operator stores a value in the object designated by the left operand. An assignment expression has the value of the left operand after the assignment, ¹¹¹⁾ but is not an lvalue.

So, what the footnote is saying is that the abstract machine needs not reread the value of a volatile lvalue to determine the value of the assignment expression.
This reading makes sense, because the text is talking about a write in an object (assignment operator), not a read.
As, e.g., in the following:

Code: [Select]

int plain_int;
int volatile volatile_int;
void f(int an_int)
{
    plain_int = volatile_int = an_int*3;
}
The abstract machine/implementation needs not read volatile_int (after the mandatory write to it) to get the value to write to plain_int (regardless whether the write to plain_int happens or not, as it's not volatile).
This can still be "surprising" as volatile_int might be a 'write 1 to clear' register (so the read value would have been different), but your examples have, in my reading, perfectly defined behaviour.

"5.1.2.3 Program execution", §2, guarantees that accessing a volatile object is regarded as a side effect.

[peter-h]

My CPU (32F417) supports unaligned access transparently, which may be why this works. It's an interesting point though; most of my instances of "buf[512]" are locally on some function stack, and I don't know if it will be aligned. Probably if buf[] is the first local variable declared, it will be, but if I have uint8_t x; before it, it may not be. Someone has just told me that members of structs are aligned on the machine word size (4 bytes in my case), unless "packed". But otherwise I have not worried about alignment, until I came across this
https://www.eevblog.com/forum/programming/packed-attribute-warning/

If one's CPU doesn't support unaligned access then there will be a huge number of gotchas all over the place, presumably.

BTW I think

Code: [Select]

__attribute__((aligned(4))) uint8_t buf[512];
is

Code: [Select]

uint8_t buf[512] __attribute__((aligned(4))) ;in GCC.

I haven't played with unions but know the idea. I just haven't needed them.

[newbrain]

But how are ensuring buffer[24] is 32-bit aligned?
Unless you tell the compiler to align buf, it could be placed in any way, casting a byte address as int32 will potentially cause a misaligned access, triggering an exception depending on the system.
The only safe way I can think off would be:

__attribute__((aligned(4))) uint8_t buf[512];

That way you could cast buffer[0, 4, 8, 12, 16, 24...] as int32 safely.

I would prefer, if C11 or later is used, to write:

Code: [Select]

_Alignas(uint32_t) uint8_t buf[512];(or using 'alignas' after including <stdalign.h>)

This would guarantee safe alignment of 0,4,8 etc offsets in the array, and non-undefined behaviour casts, and is independent of the actual alignment value of uint32_t.
But, really, I like Nominal Animal proposal better.

[newbrain]

Someone has just told me that members of structs are aligned on the machine word size (4 bytes in my case), unless "packed".

No, this is not true.
Though the standard does not impose any requirement that the padding be minimized ("6.7.2.1 Structure and union specifiers", §15, 17) apart from forbidding initial padding, members are in general aligned to their natural alignment.
See here.

As for arrays of char/(u)int8_t, they have no stricter requirement than a single char, so they can (and will) be misaligned for anything larger, as you correctly say.

[peter-h]

Interesting. I found the 512 byte USB MSB buffer which was a member of a struct, nothing aligned, so I put the align attribute on that within the struct; I take it that is allowed.

I am sort of surprised one can't do something like

uint32_t fred;
uint8_t buf[512];
fred = buf[24];

because you can shoot yourself in the foot so easily in C so why can't you shoot yourself in the foot with that? Probably because it is valid but loads just the lowest byte of fred, from buf[24].

How about

fred = &buf[24];
fred = &fred;

That's how it would be done in assembler.

My problem is not understanding pointer notation I've seen so many bugs come from that so I avoid them.

[magic]

It's not that it "just happens" to be equivalent...

• If uint8_t exists, it is exactly 8 bit wide (no padding allowed). "7.20.1.1 Exact-width integer types"
• 8 is also the minimum number of bits for any object that is not a bit field, namely CHAR_BIT. "5.2.4.2.1 Sizes of integer types <limits.h>"
• CHAR_BIT is also by definition the bit width of a char. "6.2.6 Representations of types"
• So, (unsigned) char cannot be larger than an uint8_t (as it would violate 2.) and cannot be smaller (as the minimum for CHAR_BIT is 8 ).
• Hence, if  uint8_t exists, it must be equivalent to unsigned char

There is more to a type than bit count.

The reason I care about "char equivalence" is because char is subject to special aliasing rules which make casting to/from any other type always legal. I'm not that much of a language lawyer and not 100% sure if there is any guarantee that uint8_t will work correctly for this purpose if it exists, I always find such use suspicious. Other than char, using differently typed pointers to the same object is generally UB minefield.

[newbrain]

Interesting. I found the 512 byte USB MSB buffer which was a member of a struct, nothing aligned, so I put the align attribute on that within the struct; I take it that is allowed.

I am sort of surprised one can't do something like

uint32_t fred;
uint8_t buf[512];
fred = buf[24];

because you can shoot yourself in the foot so easily in C so why can't you shoot yourself in the foot with that? Probably because it is valid but loads just the lowest byte of fred, from buf[24].

How about

fred = &buf[24];
fred = &fred;

That's how it would be done in assembler.

My problem is not understanding pointer notation I've seen so many bugs come from that so I avoid them.

The first snippet will do exactly what it says: assign the content of buf[24] (an uint8_t) to fred (a uint32_t) - i.e. fred will contain a value between 0 and 255.

The second snippet is not really meaningful, i imagine you intended something like:

Code: [Select]

fred_p = &buf[24];
fred = *fred_p;
Now what happens depends on the nature of fred_p. If it is declared as an uint32_t * you should get a warning from the compiler (at least, with sane options), and at the end fred will contain an uint32_t value, taken from buf[24..27].
If, instead it is an uint8_t *, the result will be the same as the first snippet.

EtA: And yes, pointer notation might be a bit confusing at the beginning.
Unary & is the "address of" unary operator. Eats its operand, and spits its address.
Unary * is the "indirection" operator. Eats its operand (which must be a non-void pointer) and gives you back the pointed to object.
& and * have an use also as bitwise AND, and multiplication respectively.
But, rejoice, if you were learning C++, you would also have references (using the same & character of the address operator) and r-value references using &&...

[newbrain]

There is more to a type than bit count.
[...]
I always find such use suspicious. Other than char, using differently typed pointers to the same object is generally UB minefield.

Yes, there is.
But not in this case.
uint8_t is not a type, but a typedef, i.e. an alias to an existing type ("7.20.1.1 Exact-width integer types").
That type, given all the constraints above, is bound to not have padding bits, and no trap representations.
It cannot be anything but unsigned char or an alias thereof.

[ejeffrey]

As far as I know, (u)int8_t, if it exists, is always a typedef for unsigned/signed char.  I don't know if there is a way to read the standard where it could be a distinct type that happens to have identical behavior to char, but for sure that is the way it is always actually implemented.  Systems where char is not 8 bits are not allowed to have uint8_t at all.

[peter-h]

I use & frequently; the pointer notation I still find confusing. Especially the double * at the start of this thread. Pure pointers I never use.

[magic]

That type, given all the constraints above, is bound to not have padding bits, and no trap representations.
It cannot be anything but unsigned char or an alias thereof.

But does the standard guarantee that alias analysis will treat a uint8_t pointer the same as char pointers?
Never make a blind assumption that it cannot possibly alias some uint32_t pointer in the same scope?

I don't care about representation compatibility. What I mean is that char is a special type explicitly permitted to be used in the sort of code that has been posted, but uint8_t is not defined to be so.

[coppice]

Someone has just told me that members of structs are aligned on the machine word size (4 bytes in my case), unless "packed".

If you think about it, that can't be true. It would allow doubles in structs to be misaligned, which nobody sane wants to happen.

There can be variations between compilers, but you usually get each item packed to its natural boundary if you don't specify the packing options. Beware when using arrays of structs when space is tight, as the waste can really add up. Pack will solve the space, but might degrade performance. Rearranging the order of the items in the struct can often get them packed snuggly without the performance hit.

[newbrain]

But does the standard guarantee that alias analysis will treat a uint8_t pointer the same as char pointers?

Yes, as typedef does not introduce new types, only synonyms of existing type.

[newbrain]

I use & frequently; the pointer notation I still find confusing. Especially the double * at the start of this thread. Pure pointers I never use.

Oh yes, you absolutely do use pointers.

'buf' is an array, but the name of an array in practically all contexts (Note£) is converted to a pointer to its first element.

When you write write buf[24], you are actually using a shorthand for (*(buf+24)) - see also Note# - that is, getting the value of what is pointed by buf+24. Since buf - in this context - has become a pointer to uint8_t, you are getting the value of the 24th uint8_t after the beginning of buf - see Note¤.

As for the "double" * in the OP:

Code: [Select]

*(volatile uint32_t*) &buf[24];one '*' is inside the cast.
A cast indicates to the compiler that you want to convert a value of one type to a value of another type.

The value to be converted here is &buf[24] => &(*(buf+24)) => buf+24 (as & and * are like matter and antimatter) so, a pointer to an uint_8.
The value you want to convert it to is the type inside () in the cast: a pointer (*) to a volatile uint32_t.
So, what the cast expression does is reinterpreting the bits that were a pointer of one kind, to be a pointer of different kind.

Now, you prepend '*' to the cast. '*' is, as said the indirection operator: it takes the pointer value to its right and retrieves the value of the object it's pointing to.
As the value to the right of * is a pointer to volatile uint32_t, what you get is a value of type volatile uint32_t, which you can now store in fred.


Note£ except when the operand of sizeof or & operators. So sizeof buf yields 512, and &buf a pointer to an array of 512 uint8_t .

Note# the [] operator takes two operands, and is commutative, so 24[buf] is in fact correct C and does exactly the same thing as buf[24].

Note¤ Pointer arithmetic in C add or subtracts in units equal to the size of the object pointed to. So if p is a pointer to uint8_t, when you add 1 you get the address of the next byte in memory, if q is instead a pointer to uint16_t, when you add 1 you get an address two bytes higher than q, etc. etc.

[magic]

But does the standard guarantee that alias analysis will treat a uint8_t pointer the same as char pointers?

Yes, as typedef does not introduce new types, only synonyms of existing type.

Okay, I gather you insist that on standard implementations uint8_t must be a proper typedef to one of the standard types, and this leaves only the chars because the ints are guaranteed to be wider.

That sounds fair enough and it probably is true, but as you see it took you some effort to prove. And I assure you that your proof will break down when they introduce short short int in C++31 and C33.

I guess I just don't see any sane reason to use uint8_t here.
If you know it's the same as char, simply use char.
If you require an 8 bit char, static assert on CHAR_BIT.
Most likely, you don't even require it, and with a few sizeof() here and there the code could be made more portable (and readable).

I still don't like this fad.