Author Topic: Why is loading a multi byte scalar variable from some address so complicated?  (Read 17395 times)

0 Members and 2 Guests are viewing this topic.

Offline peter-hTopic starter

  • Super Contributor
  • ***
  • Posts: 5980
  • Country: gb
  • Doing electronics since the 1960s...
I think people use uint8_t to clearly indicate that the value is used as 0-255.

And int8_t as -128 to +127 (or -127 to +127 in a lot of code); I have never used this because I am always "controlling" some hardware and 7 bits is pretty useless. Didn't stop Honeywell designing an autopilot using int8 for the control loop, with interesting results (google kfc225) ;)

Whereas a "char" is just ambiguous. I use it for storing text and stuff like that. It is less typing because a lot of standard functions expect a char and a uint8_t has to be cast with (char*) all over the place.
Z80 Z180 Z280 Z8 S8 8031 8051 H8/300 H8/500 80x86 90S1200 32F417
 

Offline DiTBho

  • Super Contributor
  • ***
  • Posts: 5090
  • Country: gb
you insist that on standard implementations uint8_t must be a proper typedef to one of the standard types, and this leaves only the chars because the ints are guaranteed to be wider.

That sounds fair enough and it probably is true, but as you see it took you some effort to prove

Yes, * right * it makes me think it was for good reason that I developed my own C-like language  :D

that C-language stuff is confusing and all rotten.
The opposite of courage is not cowardice, it is conformity. Even a dead fish can go with the flow
 

Offline newbrain

  • Super Contributor
  • ***
  • Posts: 1906
  • Country: se
Yes, * right * it makes me think it was for good reason that I developed my own C-like language  :D

that C-language stuff is confusing and all rotten.
Why, then, did you develop a C-like language instead of an Ada-like or Modula2-like language?
(just being a prick, you should know me by now...)
Nandemo wa shiranai wa yo, shitteru koto dake.
 

Offline DiTBho

  • Super Contributor
  • ***
  • Posts: 5090
  • Country: gb
Whereas a "char" is just ambiguous

"char" must be banned
casting must be banned, especially casting to/from char
char_t must  be used instead, and restricted ONLY to strings

uint8_t, uint16_t, uint32_t, uint64_t, uint128_t, uint256_t, uint512_t, ... ---> unsigned numbers
sint8_t, sint16_t, sint32_t, sint64_t, sint128_t, sint256_t, sint512_t, ... ---> signed numbers
char_t ---> strings

char_t must be part of the unsigned class
char_t must only have comparing operators { !=, ==, >, <, >=, <= }
char_t must use special uc-functions when you need to add or subtract something
            op={ +, - }           
            f(op, char_t)->(char_t):
                    -> (op,autotype<-sizeof(char_t))
                    -> char_t
            autotype and its inner operator must be automatically managed by the language
            so, char_t can be { ASCII-7bit, ASCII-8bit, something-fancy-16bit, ...}

my-C is very "nazy" about this  :D
« Last Edit: October 25, 2022, 02:51:56 pm by DiTBho »
The opposite of courage is not cowardice, it is conformity. Even a dead fish can go with the flow
 

Offline DiTBho

  • Super Contributor
  • ***
  • Posts: 5090
  • Country: gb
Why, then, did you develop a C-like language instead of an Ada-like or Modula2-like language?

This way I can recycle old projects without the need to rewrite them from scratch  :D

Practically, everything that is already MISRA{95,2000} & DO178{B,C}-level{A,B} compliant will also be  already 90% compliant with my-C

       raw -> MISRA -> DO178 -> commit: 100% C compliant, 90% my-C compliant

The remaining 10% means ... adapt the subtleties, which is always a pleasure, never a pain like converting a raw C source into a MISRA-C-compliant source

DO178{B,C}-level{A,B} adds other layers of polishing on the top of a MISRA-compliant source
The opposite of courage is not cowardice, it is conformity. Even a dead fish can go with the flow
 
The following users thanked this post: newbrain

Offline SiliconWizard

  • Super Contributor
  • ***
  • Posts: 17774
  • Country: fr
That's a reasonable rationale, although it obviously limits you. I had done some work on some evolution of C myself, bringing mainly modules and some generics, but I gave up and moved on (so far). In the end, I wasn't so sure it was worth the trouble.

One language I would consider as an inspiration could be Modula-3. The language report is available if you look a little. Pretty neat in a number of aspects. The CM3 project is back to active: https://github.com/modula3/cm3 . As is, the language is interesting but there sure are some things I would change.

Or, a "leanified" version of Ada. Of course, whatever would need to be removed is yet to be defined.

So, we're still back to C. Endlessly. ;D

 
The following users thanked this post: DiTBho

Offline ejeffrey

  • Super Contributor
  • ***
  • Posts: 4832
  • Country: us
But does the standard guarantee that alias analysis will treat a uint8_t pointer the same as char pointers?
Yes, as typedef does not introduce new types, only synonyms of existing type.
Okay, I gather you insist that on standard implementations uint8_t must be a proper typedef to one of the standard types, and this leaves only the chars because the ints are guaranteed to be wider.

That sounds fair enough and it probably is true, but as you see it took you some effort to prove. And I assure you that your proof will break down when they introduce short short int in C++31 and C33.

I guess I just don't see any sane reason to use uint8_t here.
If you know it's the same as char, simply use char.
If you require an 8 bit char, static assert on CHAR_BIT.
Most likely, you don't even require it, and with a few sizeof() here and there the code could be made more portable (and readable).

I still don't like this fad.

What fad?  Data types with an explicit well defined size and behavior?  And I don't really get what you are saying about using sizeof() instead?

(u)int8_t is preferable to char in all situations except text.  The biggest reason is that the standard allows char to be signed or unsigned.  Any time the language specifies promotion to int (which is very often) it can be zero or sign extended, it's an implementation/platform choice.  So for instance the  version of the code in the OP that does byte assembly for endian conversion:

Code: [Select]
uint32_t fred = buf[24] | (buf[25]<<8) | (buf[26]<<16) | (buf[27]<<24);

is wrong if buf is defined as char[] instead of uint8_t[]. Yet it will behave correctly on some platforms while failing on others.  This is not some theoretical "allowed by the standard but never exists in practice" detail, ARM usually uses unsigned char and x86/amd64 uses signed.  GCC has an option to change it because a lot of code is written expecting one behavior and gets ported to platforms with the other.

You can write out "signed char" or "unsigned char" everywhere, but (u)int8_t is shorter and more common.  Even for text data people often do  comparisons that could depend on the sign behavior so it would be ideal if all char variables had a defined ranges, but they don't and all the standard library string handling functions use char in their signatures.  All basic ASCII characters are positive and integer value comparison is mostly meaningless for non-ASCII data, so it's not a big deal in practice, so maybe that's not a big deal anyway.

If you are strictly using a generic buffer and always casting to another type before manipulating the data, then obviously the type doesn't really matter.  But if you ever might want to treat the data as an array of bytes, it makes sense to use a type that has defined behavior.
 
The following users thanked this post: peter-h, newbrain

Offline DiTBho

  • Super Contributor
  • ***
  • Posts: 5090
  • Country: gb
is wrong if buf is defined as char[] instead of uint8_t[]. Yet it will behave correctly on some platforms while failing on others.  This is not some theoretical "allowed by the standard but never exists in practice" detail, ARM usually uses unsigned char and x86/amd64 uses signed.  GCC has an option to change it because a lot of code is written expecting one behavior and gets ported to platforms with the other.

Indeed on hppa2 it fails.
The opposite of courage is not cowardice, it is conformity. Even a dead fish can go with the flow
 

Offline peter-hTopic starter

  • Super Contributor
  • ***
  • Posts: 5980
  • Country: gb
  • Doing electronics since the 1960s...
This is all really interesting.

The stdlib string compare funcs, expecting "char", must do weird things for char codes > 127. You will still get the right result for equality but the < or > will be unreliable.

Quote
uint32_t fred = buf[24] | (buf[25] << 8 ) | (buf[26]<<16) | (buf[27]<<24);
is wrong if buf is defined as char[] instead of uint8_t[]. Yet it will behave correctly on some platforms while failing on others. 

That's another lesson for me. I ought to check if all the buffers are uint8_t.

« Last Edit: October 26, 2022, 09:24:15 am by peter-h »
Z80 Z180 Z280 Z8 S8 8031 8051 H8/300 H8/500 80x86 90S1200 32F417
 

Offline DiTBho

  • Super Contributor
  • ***
  • Posts: 5090
  • Country: gb
That's another lesson for me

People who gain the constant support of others on the forums and then still produce shit code.
That's *THE* problem with opensource.
Ignorant and lazies usually never grow, because they can always gain the constant support of others.
The opposite of courage is not cowardice, it is conformity. Even a dead fish can go with the flow
 

Offline peter-hTopic starter

  • Super Contributor
  • ***
  • Posts: 5980
  • Country: gb
  • Doing electronics since the 1960s...
Quote
People who gain the constant support of others on the forums and then still produce shit code.
That's *THE* problem with opensource.
Ignorant and lazies usually never grow, because they can always gain the constant support of others.

Who are you referring to?

If you mean me, well thank you, but I am working totally alone, with nobody to ask for help. So I use forums quite a bit. I learnt C from almost zero, 2 years ago, after decades of assembler. The fact that my posts often generate a lot of responses indicates that these are real issues which catch people out.

What regular expression would pick up
Code: [Select]
char * [*] ?
« Last Edit: October 26, 2022, 11:27:22 am by peter-h »
Z80 Z180 Z280 Z8 S8 8031 8051 H8/300 H8/500 80x86 90S1200 32F417
 

Offline Nominal Animal

  • Super Contributor
  • ***
  • Posts: 8349
  • Country: fi
    • My home page and email address
The stdlib string compare funcs, expecting "char", must do weird things for char codes > 127.
Well, look at the <ctype.h> isclass(code) functions.  Given char ch, you do NOT call them isclass(ch), you explicitly must use isclass((unsigned char)ch) instead.  So, it's not "weird".  The only reason the interface specifies an int is because they have to work for all character codes, plus EOF.

Besides, the C standard itself says that <string.h> functions "shall interpret [each character] as if it had type unsigned char" (e.g. C11 7.24.1p3).

In other words, practical implementations of current C libraries internally treat char for strings as unsigned char; it's just that the API is fixed to char.

Quote
uint32_t fred = buf[24] | (buf[25] << 8 ) | (buf[26]<<16) | (buf[27]<<24);
is wrong
, period.  It relies on integer promotions, which is not reliable if buf elements are of a signed type.  If any of them have a negative value, then the high bits of fred will be set, because uintN_t types by definition use twos complement format.

The correct expression, and the one you really, really should be using, is
Code: [Select]
    uint32_t  fred =  (uint32_t)(buf[24])
                   | ((uint32_t)(buf[25]) << 8)
                   | ((uint32_t)(buf[26]) << 16)
                   | ((uint32_t)(buf[27]) << 24);
The reason is twofold:
  • By explicitly casting the data element to a sufficiently large unsigned integer type you ensure the correctness of the result, instead of relying on implicit integer promotions.  Although you could use some other type than the result of the entire expression, using the same type avoids any surprises.
     
  • Explicitly writing out the logic, even when not strictly required by the language syntax, reduces human cognitive load.  The above code is immediately clear, completely without any ambiguities.  Its only downside is verbosity, which can be annoying.  However, that verbosity has a purpose – clarity, minimum cognitive load, minimum number of surprises – which more than compensates for it.

Feel free to disagree, anyone, but relying on automatic type promotions and incorrect assumptions in C, leads to annoying bugs.  The most common example is the assumption that int and pointers are the same size, and that given void *p, then (void *)((int)p) == p.  This particular bug is common enough that most recognize it in this form, and yet most use int for array index variables, and end up wondering when their code suddenly either locks up (neverending loop) or produces garbage (code loops over only part of the data) given sufficient amounts of data.

In short, I've seen time and time again how avoiding having to write the full expressions lead to annoying bugs.  In the balance, you have known correct expressions on one side, and your preference for brevity in the other.  Do be honest about what really matters to you.



In this and related threads, I've suggested to use the explicit type appropriate for how the data is used.  Here, it means that if you do intend to alias the buffer contents by other types –– i.e., that it actually consists of fields of varying types and sizes (in bytes) ––, you use unsigned char.

unsigned char is a special type. Since C11 (6.2.6.1p3, p4), the standard guarantees that unsigned char is a binary type of CHAR_BIT bits with no padding bits; that it can describe values between 0 and 2CHAR_BIT-1, exactly; no more, no less.  Any object of size n bytes can be copied to an array of unsigned char [n] to obtain its object representation.  In other words, the C11 standard says that unsigned char is the type you can use for manipulating the representation of any object.

Now, if the data is consumed in aligned 32-bit units, and most fields do not cross a 32-bit boundary, then the better choice is uint32_t buf[bytes/4]; instead.  Remember that you can use a simple cast expression, ((unsigned char *)buf) to access the same buffer as if you had declared it as unsigned char buf[bytes];, so this is a choice whose indirect effects –– it will be aligned, up to 32-bit access via uint32_t of elements that do not cross a 32-bit boundary is trivial, and so on –– should be considered carefully.
 

Offline peter-hTopic starter

  • Super Contributor
  • ***
  • Posts: 5980
  • Country: gb
  • Doing electronics since the 1960s...
Quote
By explicitly casting the data element to a sufficiently large unsigned integer type you ensure the correctness of the result, instead of relying on implicit integer promotions.  Although you could use some other type than the result of the entire expression, using the same type avoids any surprises.

May I clarify that

Code: [Select]
uint32_t  fred =  (uint32_t)(buf[24])
                   | ((uint32_t)(buf[25]) << 8)
                   | ((uint32_t)(buf[26]) << 16)
                   | ((uint32_t)(buf[27]) << 24);

is equivalent to the above without the uint32_t casts if buf is type uint8_t ?

I also tend to AND values with 0xff e.g.

Code: [Select]
                uint32_t fact_size = xxxxxxx
                uint8_t ssa_buf[512];

ssa_buf[4]=(fact_size & 0x000000ff);
ssa_buf[5]=(fact_size & 0x0000ff00) >> 8;
ssa_buf[6]=(fact_size & 0x00ff0000) >> 16;
ssa_buf[7]=(fact_size & 0xff000000) >> 24;

fact_size is a uint32_t but the above shifts should leave all bits other than the byte of interest at zero.
Z80 Z180 Z280 Z8 S8 8031 8051 H8/300 H8/500 80x86 90S1200 32F417
 

Offline Nominal Animal

  • Super Contributor
  • ***
  • Posts: 8349
  • Country: fi
    • My home page and email address
May I clarify
It is not a clarification.  It is an admission that you'd rather make an assumption than write a few more characters.  That's fine; it's perfectly acceptable business decision.  But don't make it sound like it is anything different than an assumption to avoid having to write the longer expressions.

      ssa_buf[4]=(fact_size & 0x000000ff);
Byte masking is only useful if the code will work even when individual elements of ssa_buf can be larger than an 8-bit byte, but still have the same semantics (i.e., the low 8 bits contain the information, even when the array elements themselves are larger than that).

If it does not, then the byte masking is not only superfluous, it is misleading.

(Typically, no extra code will be generated if any optimizations are enabled, so the main drawback is how it can mislead us humans.)
 

Offline peter-hTopic starter

  • Super Contributor
  • ***
  • Posts: 5980
  • Country: gb
  • Doing electronics since the 1960s...
Quote
It is not a clarification.  It is an admission that you'd rather make an assumption than write a few more characters.  That's fine; it's perfectly acceptable business decision.  But don't make it sound like it is anything different than an assumption to avoid having to write the longer expressions.

What I was getting at is the definition of C.

If you do

fred = buf[25] << 8

then

- buf[25] is extracted as a uint_8 byte
- promoted to a int32 (on arm32/gcc) - the "integer promotion" thingy
- shifted left 8, so the original byte is now in bits 9-15 of the int32
- cast into the destination type (uint32)
- loaded into the destination


Z80 Z180 Z280 Z8 S8 8031 8051 H8/300 H8/500 80x86 90S1200 32F417
 

Offline Nominal Animal

  • Super Contributor
  • ***
  • Posts: 8349
  • Country: fi
    • My home page and email address
Consider this function I often use to parse a command-line argument of type int:
Code: [Select]
#include <stdlib.h>
#include <string.h>
#include <errno.h>

int  parse_int(const char *from, int *to)
{
    const char *ends;
    long  val;

    if (!from || !*from)
        return -1;  /* NULL or empty string */

    ends = from;
    errno = 0;
    val = strtol(from, (char **)&ends, 0);
    if (errno || ends == from)
        return -1;  /* Error in conversion */

    while (*ends == '\t' || *ends == '\n' || *ends == '\v' ||
           *ends == '\f' || *ends == '\r' || *ends == ' ')
        ends++;
    if (*ends)
        return -1;  /* Garbage at end of string */

    if ((long)((int)val) != val)
        return -1;  /* ??? */

    if (to)
        *to = val;
    return 0;
}
The question is, given that we have long val, what use does the expression ((long)((int)val) != val) have?

The answer is, because the C standard says that a cast to a numeric type limits the range and precision of the cast expression to that of the cast type, that aforementioned expression is true if and only if val cannot be represented by an int.

C casts are powerful, because they convey specific limitations or requirements for a value, without generating any particular machine code (like a function call or anything like that) for it, only change the code they generate related to the cast expression itself.

(Just consider how you would have checked against overflow there yourself.  INT_MIN and INT_MAX from <limits.h>, perhaps?  That would indeed be more readable, but would also generate additional code.)

Okay, but what does any of this have to do with the thread at hand?

If you were to explore with the various suggestions posted at Compiler Explorer, you'd see that I'm pushing for unambiguous code that compiles to acceptable machine code.  Casts and their effects are a perfect example of that.  (I am not going for any "perfect" or "best", because I'm perfectly willing to trade a cycle here, a dozen there, for readable, unambiguous, easily maintained –– if verbose –– code.)

And okay, I might be proselytizing a bit about how we should not be afraid of being verbose when it is useful; that being succinct by relying on not very well known implicit behaviour (like the exact integer promotion rules) is not optimal.  I'd go as far as saying it is hacky.  Feel free to disagree, however; I've hopefully made the reasons for these statements clear.  (Other than making a list of links to example bugs created when people rely on assumptions and incorrect assumptions, but that would be too depressing for me  :'(.)
 
The following users thanked this post: newbrain

Offline newbrain

  • Super Contributor
  • ***
  • Posts: 1906
  • Country: se
May I clarify that

Code: [Select]
uint32_t  fred =  (uint32_t)(buf[24])
                   | ((uint32_t)(buf[25]) << 8)
                   | ((uint32_t)(buf[26]) << 16)
                   | ((uint32_t)(buf[27]) << 24);

is equivalent to the above without the uint32_t casts if buf is type uint8_t ?
The above code, without the cast and with an uint8_t buf, is another case where Undefined Behaviour rears its ugly head.

You yourself analysed correctly what happens in a similar case:
fred = buf[25] << 8

then

- buf[25] is extracted as a uint_8 byte
- promoted to a int32 (on arm32/gcc) - the "integer promotion" thingy
- shifted left 8, so the original byte is now in bits 9-15 of the int32
- cast into the destination type (uint32)
- loaded into the destination

In the code without the cast, this is fine for values 0, 8, and 16 of the shift amount (and an unsigned type for buf[]), but falls apart when when we shift by 24 a value in [0..256).
The operands of the << operator are now of type int (as you say!), and according to "6.5.7 Bitwise shift operators", §3 and 4:
Quote from: C11, emphasis mine
The integer promotions are performed on each of the operands. The type of the result is that of the promoted left operand.[...]
The result of E1 << E2 is E1 left-shifted E2 bit positions; [...]
If E1 has a signed type and nonnegative value, and E1 × 2E2 is representable in the result type, then that is
the resulting value; otherwise, the behavior is undefined.

So, no the code is not equivalent and elicits UB for all values of buf[27] > 127.

Will it work? Probably, on all practical architectures, but it's still not conforming and UB.
A (too) smart compiler might notice the UB and produce optimized code that does not give the expected result.
« Last Edit: October 26, 2022, 03:00:04 pm by newbrain »
Nandemo wa shiranai wa yo, shitteru koto dake.
 

Offline peter-hTopic starter

  • Super Contributor
  • ***
  • Posts: 5980
  • Country: gb
  • Doing electronics since the 1960s...
So I got away with the x[y] << 24 because the 32 bit value (in my application it is stuff like code size; always< 1MB on a 32F417) cannot ever be bigger than 3 bytes. In fact the code would work for any 32 bit value which has bit 31 = 0.

But I went through the code in my project and a lot of cases omit that cast, yet seem to work, in code written by others, including ST, and which ought to have failed because the MS byte will sometimes be > 127. So I don't get it.

Anyway I searched for
>>24
>> 24
and edited in that (uint32_t) and will now do a ton of testing to make sure it still works.

Unbelievable!

Well, the mod has broken various things...
« Last Edit: October 26, 2022, 03:26:41 pm by peter-h »
Z80 Z180 Z280 Z8 S8 8031 8051 H8/300 H8/500 80x86 90S1200 32F417
 

Offline Nominal Animal

  • Super Contributor
  • ***
  • Posts: 8349
  • Country: fi
    • My home page and email address
If one wanted to be really, really explicit, then
Code: [Select]
uint32_t  fred =  (uint32_t)((uint8_t)(buf[24]))
               | ((uint32_t)((uint8_t)(buf[25])) << 8)
               | ((uint32_t)((uint8_t)(buf[26])) << 16)
               | ((uint32_t)((uint8_t)(buf[27])) << 24);
In this case, each array element is first cast into an 8-bit unsigned integer type.  Then, each is cast to a sufficiently large unsigned integer type (best use the same type as the result of the entire expression, although fast unsigned types –– uint_fast32_t here –– would also work equally well), and then shifted to their final position.  The four are then binary-OR'd together to get the final result.

When the cast is to an unsigned integer type, the conversion is done using modulo arithmetic (C11 6.3.1.3p2), regardless of whether the original integer type is signed or unsigned.  Thus, a cast to uint8_t type is effectively equivalent to a binary AND with 255.  Similarly, a cast to uint16_t is effectively equivalent to a binary AND with 65535, and a cast to uint32_t to a binary AND with 4294967295.  (The same applies to all unsigned integer types: unsigned char and & UCHAR_MAX, unsigned short and & USHRT_MAX, unsigned int and & UINT_MAX, unsigned long and & ULONG_MAX, and if supported, unsigned long long and & ULLONG_MAX.)
The key to remember is that it is modular and not saturating conversion.

Why "effectively equivalent"?  The practical result in terms of the C abstract machine are the same, but the cast is often easier for the compiler to optimize than the binary AND.

So I got away with the x[y] << 24 because the 32 bit value (in my application it is stuff like code size; always< 1MB on a 32F417) cannot ever be bigger than 3 bytes. In fact the code would work for any 32 bit value which has bit 31 = 0.
You "got away" with it, because the elements are of unsigned integer type, and your compiler uses modular arithmetic for left shift on the signed int type.

First, each element gets promoted to int (but never become negative; they'd be promoted to unsigned int if the value could not be represented by int – the C integer promotion rules are quite explicit).  Then, each element is shifted left.  If the seventh bit in the highest byte is set, we invoke Undefined Behaviour.

To repeat what Newbrain already wrote above, but in a different form just so that readers here understand (because it is kinda important):

Consider the case where int E1 = 128, and we do result = E1 << 24.  Technically, since INT_MAX = 2147483647 on your architecture, E1 << 24 is Undefined Behaviour, because:
Quote from: C11 6.5.7p4
The result of E1 << E2 is E1 left-shifted E2 bit positions; vacated bits are filled with zeros. If E1 has an unsigned type, the value of the result is E1 × 2E2, reduced modulo one more than the maximum value representable in the result type. If E1 has a signed type and nonnegative value, and E1 × 2E2 is representable in the result type, then that is the resulting value; otherwise, the behavior is undefined.
We fall in the otherwise, the behaviour is undefined case, because E1 is signed (but nonnegative value), but E1×224 = 231 = INT_MAX+1 > INT_MAX.

However, there is so much code out there relying on the same behaviour that you are, that all current C compilers behave as if E1 was cast to the corresponding unsigned type first, then the shift applied, and finally the result cast to the original signed type.  So, I wouldn't be too worried about a compiler generating wrong machine code (compared to the obvious programmer intent) here, although a strict reading of the C11 standard says it could do anything it wants, including produce nasal daemons.  If they generate wrong machine code for this, a lot of other existing code will miscompile too, and not work; compiler users are quite unhappy about such changes, and tend to switch to using a different compiler instead.

Anyway, I don't like the subtext of "getting away with it" here, because really, it is all about what the logic of your expressions is based on.
Put bluntly, I don't think you are "getting away with" something, I just think you are relying on things without knowing you are relying on them, and want to inform you of exactly what you are relying on when using such expressions.  (It is not wrong per se to rely on them, but for long-term maintenance et cetera, you do want to document them at least.)

This is also exactly the reason why I sometimes insist on technically incorrect/incomplete, but intuitively constructive, analogs and explanations.
The worth of intuitively understanding exactly what you are basing your expectations on when writing such seemingly simple expressions in C is, in my opinion, very high.  The same applies to pointers and aliasing, and unions and type punning as well.  It is what leads to better code, in my opinion.
« Last Edit: October 26, 2022, 03:35:31 pm by Nominal Animal »
 

Offline peter-hTopic starter

  • Super Contributor
  • ***
  • Posts: 5980
  • Country: gb
  • Doing electronics since the 1960s...
Thank you - it explains why it works.

Now I will restore a dozen files from yesterday's backup :)

Quote
a lot of other existing code will miscompile too

Yeah - I found a lot of examples, and not from me either.

Quote
First, each element gets promoted to int (but never become negative; they'd be promoted to unsigned int if the value could not be represented by int – the C integer promotion rules are quite explicit).  Then, each element is shifted left.  .

That suggests to me that the (uint32_t) cast is potentially needed only on the byte being shifted left by 24.

Quote
If the seventh bit in the highest byte is set, we invoke Undefined Behaviour

How does the compiler know if bit 7 will be 1, at compile time?
« Last Edit: October 26, 2022, 05:30:29 pm by peter-h »
Z80 Z180 Z280 Z8 S8 8031 8051 H8/300 H8/500 80x86 90S1200 32F417
 

Offline newbrain

  • Super Contributor
  • ***
  • Posts: 1906
  • Country: se
That suggests to me that the (uint32_t) cast is potentially needed only on the byte being shifted left by 24.
[...]
How does the compiler know if bit 7 will be 1, at compile time?
1. Yes, the other shifts are safe - that's why I singled that out.

2. It does not and cannot. The UB is potential depending on the runtime data.
Still it might be able to understand that there a potential UB, where the standard makes no requirement on the emitted code behaviour.
In this specific example, no big deal, but in others the compiler can (and will) take shortcuts so the code works best for defined case, and anything can happen in the UB case, giving a "wrong" result (the result cannot really be "wrong", as any result is right when UB comes into play).

If one is absolutely certain that buf[27] will never, ever, exceed 127 then you are fine. But:
a. it's quite difficult to prove (in general)
b. this kind of assumptions should be well commented/documented
So, in the end, better write compliant code. It's less work, manual and mental.
Nandemo wa shiranai wa yo, shitteru koto dake.
 

Offline peter-hTopic starter

  • Super Contributor
  • ***
  • Posts: 5980
  • Country: gb
  • Doing electronics since the 1960s...
So if I have say

Code: [Select]
uint32_t data=buffer1[i]|(buffer1[i+1]<<8)|(buffer1[i+2]<<16)|(buffer1[i+3]<<24);
it is more correct to have

Code: [Select]
uint32_t data=buffer1[i]|(buffer1[i+1]<<8)|(buffer1[i+2]<<16)|((uint32_t)buffer1[i+3])<<24);
There are many cases where the former way is used e.g. in handling of an IP (pre-IPV6 obviously; that is a lot more involved), whose bytes can and are just about any value. Yet, this code is widely used and it works.
Z80 Z180 Z280 Z8 S8 8031 8051 H8/300 H8/500 80x86 90S1200 32F417
 

Offline DiTBho

  • Super Contributor
  • ***
  • Posts: 5090
  • Country: gb
Who are you referring to?

those to whom this patch was aimed
Quote
fbdev/sis: use explicitly signed char
the same problem, over and over again, after spending hours on forums and mailing lists telling people the difference between signed and unsigned.
The opposite of courage is not cowardice, it is conformity. Even a dead fish can go with the flow
 

Offline Nominal Animal

  • Super Contributor
  • ***
  • Posts: 8349
  • Country: fi
    • My home page and email address
Quote
If the seventh bit in the highest byte is set, we invoke Undefined Behaviour
How does the compiler know if bit 7 will be 1, at compile time?
Consider a hardware architecture that has an arithmetic shift left, i.e. multiplication by a power of two, that either saturates or wraps around to a positie value.  There could be a non-C reason why an architecture has such an instruction.  We already have binary and arithmetic shifts right, where binary shift rotates in zeroes, and arithmetic shift copies of the most significant bit on twos complement architectures.

In that case, a compiler would be well within the standard and use that instruction.  The end result would be that the sign bit of the int would always be zero, and assuming 32-bit architecture, so would be the most significant bit of the result.

There are many cases where the former way is used e.g. in handling of an IP (pre-IPV6 obviously; that is a lot more involved), whose bytes can and are just about any value. Yet, this code is widely used and it works.
Yes, sure.  My point is that it works when a specific set of assumptions are fulfilled, only.

Because relying on unstated and undocumented assumptions leads to bugs –– just consider the "ints are the same size as pointers" debacle we had to deal with when porting code to first LP64 architectures! ––, I'm trying to show you and others what those assumptions are.

This is part of my nefarious plan.

You see, the next step in my nefarious plan is to convince you and others to document these assumptions, centrally, in a README-like file, and with comments referencing each such item in that set wherever applicable (perhaps at the beginning of each source file).  This reduces the long-term maintenance cost, and makes combating bit-rot (as assumptions change as time passes, again recall sizeof(int) == sizeof(void *) which used to be the rule) much easier and lower cost.

At some point, at least some of us will port at least some of our embedded ILP32 code to Aarch64 (ARMv8-a) or later 64-bit architectures.  Sure, GCC and Clang do support ILP32 ABI at compile/link time, but that'll bite you if you ever have more than 4GiB address space: then, you really do have to switch to LP64.  Why assume you'll only ever work in ILP32 in C, and limit yourself?  It is extremely easy to ossify oneself and stop learning and adapting, if you start choosing the "least effort" way.

We do know from the Cobol folks that as long as you are good and experienced enough, there will be some demand.  So as a business decision, it is a valid choice to focus on ILP32 only, though.
« Last Edit: October 26, 2022, 07:44:33 pm by Nominal Animal »
 

Offline peter-hTopic starter

  • Super Contributor
  • ***
  • Posts: 5980
  • Country: gb
  • Doing electronics since the 1960s...
Quote
Consider a hardware architecture that has an arithmetic shift left, i.e. multiplication by a power of two, that either saturates or wraps around to a positie value.

I have never (significantly) programmed arm32 in assembler but normally CPUs have a shift left with a zero fed in from the right. A 1 fed in from the right would be completely pointless. Then you have rotation (usually via the carry bit) instructions but one would never use those for a left shift because one needs to clear the carry at each step. But surely in the modern context a << 24 would be done with either a barrel shifter or with byte extraction.

How about a right shift i.e. uint8 >> 24. Is there not a similar problem there?

I will for sure document this issue; I have an extensive hardware/software design document for this project.
Z80 Z180 Z280 Z8 S8 8031 8051 H8/300 H8/500 80x86 90S1200 32F417
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf