2 byte value into 2 separate 1 byte values

[TheCalligrapher]

Regarding alignment; most proper CPUs give ALIGNMENT ERROR interrupt (for example, busfault), making it easy to see what happened, why, and where, and fix it. Instead, a mistyped bitshift somewhere is completely hidden and only causes some certain parameter or field to act strangely. Almost impossible to debug; you may not even know you have a problem.

Wow... Defending an error prone hardware-level hack on the basis that this hack is likely to fail hard, thus alerting the author that his error-prone hardware-level hack actually lived up to its name an ended up with an error!

It is like building cars with wheels that fall off. You know "It doesn't go far, but at least you always know what the problem is!"

I'm having hard time convincing myself that the above quoted author is not intending this as a tongue-in-cheek statement.

[PlainName]

The problem with the arithmetic version, whether based on / % * or << >> | &, doesn't matter, is that it is error prone manual work.

Pretty much all programming is error-prone manual work! We know that = is often used when == is intended, but we don't use hardware-level hacks instead of assignments or comparison. This (the merge/separation of bytes) is so common that it should be easy to spot a mistype, but even easier to use a macro of inline function. And this is also why we have unit tests - if you're not sure your code does what you mean it to do, you can just test it before locking it and never having to give it another thought.

[SiliconWizard]

Agree with you two here. With that reasoning, a simple incrementation is also error-prone and should be avoided. Indeed, "n += 1;"... but you could mess it up and write "n += 2;" or "n -= 1;" instead.

All programming is error-prone manual work. Actually, if some programmer couldn't get a basic arithmetic operation right, I would have little reason to trust any of their work.

[AndersJ]

Agree with you two here. With that reasoning, a simple incrementation is also error-prone and should be avoided. Indeed, "n += 1;"... but you could mess it up and write "n += 2;" or "n -= 1;" instead.

Why are increments written in that error prone and incomprehensible way?
Why not write N = N + 1?
Avoids all confusion, is readable and impossible to misunderstand.

[fourfathom]

Why not write N = N + 1?

Because "N" is likely to actually be  "someRidicuouslyLongVariableName", and I'd rather not have to type that twice.  Anyway, I'll use the "++" operator if it's available .

As for the two separate byte problem, I prefer the shift and mask technique, but then I'm a hardware engineer.

[PlainName]

That's a good question. Ignoring stuff like saving space (which is actually useful if names are long), I think the added benefit is that they are easily differentiated. For isntance,

ThisVariable = ThatVariable + 2;

You gotta  actually read these to figure out that it's not += 2, and generally we don't read but just look at the shapes. Carrying on...

ThisVariable++;

Straight away we know it's a simple increment, not add 2 or 3 or whatever.

Perhaps not definitive reasons, but I think they add enough to allow them without resorting to 'lazy typists' stuff

[golden_labels]

All programming is prone to error, but there is no reason to make bugs more likely.

In this situation I am less concerned about mistyping code. More about having to take care about data types and their ranges. If not that most architectures use two’s complement and compilers use the simplest approach to implementing arithmetic, accidently masking errors in arithmetic, huge portion of C code would explode.

[brucehoult]

The alternative solution, based on arithmetic, may be be valid, but may as well contain a bug.

This is a very good point. Personally, I worked with hand-maintained bit shift fests until one day I hunted for a bug for hours; a bug which was caused by mistyping < instead of <<. Then I started thinking, there has to be a better way.

Having even *one* single unit test with well designed random-looking input values and the expected output will catch < instead of <<, or shifting by the wrong amount, or extracting field 3 into output 2, or or or ...


Thirty years ago I thought correct programs could be written by using a bondage-and-discipline language and strictly following type checking and not using "unsafe" constructs and so forth. I looked down on such hacks as using macros instead of templates, or #defines and conditional compilation, or running sed over source code, or writing little bits of perl to generate C source, or doing hacky address arithmetic, or using an untyped or dynamically typed language. If you couldn't do something strictly in-language in Pascal or Ada or OCaml then you shouldn't be doing it.

I was wrong.

Even following all the disciplines in those languages, there are far more errors you can still make than errors they prevent. And they put a whole lot of barriers in the way of effective programming.

It's much better to write things in the way that lets you have the smallest amount of hand-written code. For example, if you're tempted to manually copy&paste&edit 20 copies of something -- totally evil! -- because you can't make that into a function or it's beyond the capabilities of macro processing (or your language doesn't have macros) *don't* *do* *that*. Automate the copying and editing using perl or sed so that you can fix a bug in the code being duplicated in one place, or so that you can change the way you edit it.

Put unit tests in place to check that the end result works as expected. Add preconditions and postconditions to functions and loops.


It's basically impossible to make code correct by design -- at least in a normal commercial setting. It's different if you're programming medical or aerospace or nuclear equipment where they are happy to spend 100x more per line of code. But in a normal setting, the only way to ensure code is correct is testing, and ONCE YOU'RE TESTING APPROPRIATELY it makes very little difference what programming language or technique you use.

[Siwastaja]

It's much better to write things in the way that lets you have the smallest amount of hand-written code. For example, if you're tempted to manually copy&paste&edit 20 copies of something -- totally evil! -- because you can't make that into a function or it's beyond the capabilities of macro processing (or your language doesn't have macros) *don't* *do* *that*

This is EXACTLY what I was saying above. Apparently others than brucehoult and golden_labels did not get it.

No, I strongly disagree with the others. If you have a nicely designed struct with 57 variables holding your state and you are constructing each of the 57 variables by writing 57 LoC each with various combinations of casting, bitshifting and masking, you are really doing it wrong. Yes, programming requires careful work, but repeated manual copy-paste-like (doesn't matter if you avoid the actual copy-pasting and rewrite manually) should still be avoided. Human brain is good in creative work, but does a lot of mistakes (say, several %) when repeating mechanical "simple" work. This is why automation exists.

Yes, you can solve the problem of being error-prone by rigorously always writing complete unit tests for such construction/deconstruction functions, but this is a lot of work just to test a mechanical piece which only exists due to lack of suitable language features. Also, unit testing does not easily reveal portability issues because arithmetic on C is non-portable (unless you Do It Right, but you can't test against this just by simple unit tests. Maybe a good static analysis tool is available, I don't know, or you can automate running unit tests on gazillion compilers and platforms...)

If type-punning packed struct through union - a hack that exists on the borderline between the "standard C" and "real-world, constrained C" - is out of question and the arithmetic way is the only acceptable, there is really no other option than to auto-generate the structs and the parser from a common description language. This is an actually useful real-world solution I have nothing against. On microcontrollers and other well-definited restricted environments where endianness is fixed and all compilers used support packed attribute, I go the most writable, most readable, most maintainable, least risky way of using the structs directly, but I indeed do accept it isn't a generic solution either. Call it a "hack" if you wish, I don't care.

Finally, the whole Internet works based on C code sharing packed structs like this, and it works, given the programmer knows what they are doing, and what the limitations are. (For example, endianness conversion macros are used.)

And yes, I'm serious about "fail hard". The "hacky" way has approximately 2-4 possible problems (padding, alignment, endianness...) most of which are either trivially dealt with (just one correctly placed attribute needed!), or cause serious, easily noted error (instant bus fault). The arithmetic way seems easy on surface but has more possible traps, not limited to typing errors but also operator precedence, integer promotion rules, numerical ranges, implicit casting, all the implementation specified rules like signed integer overflow, and so on and so on. This is so risky that unit testing for every input combination is highly recommendable.

I would never trust a programmer who claims they are so excellent they make zero mistakes in large sets of "simple" arithmetic operations. Take note, SiliconWizard. I would have expected better commentary from you with your excellent track record of your posts.

Actually why I do recommend the arithmetic version for beginners is to expose them to all the problems, it's highly likely they don't get it right the first time. I didn't, I hit different problems for years until finally nailing it, and after nailing it all that's left are simple accidents. Because arithmetics in C can't be avoided, better expose oneself to it.

Arithmetic version is not guaranteed to be portable either because in original C standard, type widths are implementation specified, and internal implicit conversions convert into these types, number literals are of varying widths... So you need to exactly do the right thing with explicit casting. So portability comes from careful work and understanding. Having seen some portability problems, I'd hazard a guess most of the actual portability problems are related to type handling in arithmetic; NOT "low-level hacks". Latter are typically not everywhere but constrained somewhere, easily dealt with.

Sadly, I think there is no generic solution.

What I don't like in this reoccurring discussion is the mental dishonesty of double stardards. If the programmer makes a mistake in the direct struct usage pattern, it's a proof of the pattern being unusable and dangerous. But if the programmer makes a mistake with the arithmetic solution, it's just a problem of the programmer not being careful enough, and solution is to be more careful and not to make any mistakes.

[brucehoult]

It's much better to write things in the way that lets you have the smallest amount of hand-written code. For example, if you're tempted to manually copy&paste&edit 20 copies of something -- totally evil! -- because you can't make that into a function or it's beyond the capabilities of macro processing (or your language doesn't have macros) *don't* *do* *that*

This is EXACTLY what I was saying above. Apparently others than brucehoult and golden_labels did not get it.

No, I strongly disagree with the others. If you have a nicely designed struct with 57 variables holding your state and you are constructing each of the 57 variables by writing 57 LoC each with various combinations of casting, bitshifting and masking, you are really doing it wrong. Yes, programming requires careful work, but repeated manual copy-paste-like (doesn't matter if you avoid the actual copy-pasting and rewrite manually) should still be avoided. Human brain is good in creative work, but does a lot of mistakes (say, several %) when repeating mechanical "simple" work. This is why automation exists.

Here's an example of what I'm talking about, using that task of extracting fields from a stream of bytes as an example.

I'm assuming the fields can be any size from 1 to 32 bits, are little-endian, and arbitrarily aligned on bit boundaries.

Rather than ad-hoc extracting each field I'd make a macro (or, better, inline function if you can force your compiler to reliably do that):

Code: [Select]

#include <stdint.h>
#include <stddef.h>

// max size 32 bits                                                                                                                 
#define EXTRACT_FIELD(FIELD, BASE, OFFSET, SIZE) \
do { \
  size_t byteOff = (OFFSET)/8, bitOff = (OFFSET)%8; \
  char *CHAR_BASE = (char*)(BASE); \
  uint32_t val = CHAR_BASE[byteOff] >> bitOff; \
  if (bitOff+SIZE >  8) val |= (uint32_t)(CHAR_BASE[byteOff+1]) << ( 8-bitOff); \
  if (bitOff+SIZE > 16) val |= (uint32_t)(CHAR_BASE[byteOff+2]) << (16-bitOff); \
  if (bitOff+SIZE > 24) val |= (uint32_t)(CHAR_BASE[byteOff+3]) << (24-bitOff); \
  if (bitOff+SIZE > 32) val |= (uint32_t)(CHAR_BASE[byteOff+4]) << (32-bitOff); \
  FIELD = val & ((1<<SIZE)-1);                                          \
 } while (0);

int foo(char *p){
  int r;
  EXTRACT_FIELD(r, p, 40, 8);
  return r;
}

int bar(char *p){
  int r;
  EXTRACT_FIELD(r, p, 10, 8);
  return r;
}

If there are bugs in the macro (maybe!), at least they can be fixed in one place. The same if you come up with a more efficient scheme.

The code generated is not too bad with -O1.

With RV32 gcc cross-compiler:
 

Code: [Select]

00000000 <foo>:
   0:   00554503                lbu     a0,5(a0)
   4:   8082                    ret

00000006 <bar>:
   6:   00154783                lbu     a5,1(a0)
   a:   8789                    srai    a5,a5,0x2
   c:   00254503                lbu     a0,2(a0)
  10:   051a                    slli    a0,a0,0x6
  12:   8d5d                    or      a0,a0,a5
  14:   0ff57513                andi    a0,a0,255
  18:   8082                    ret

Native system compiler (Clang) on my Mac:

Code: [Select]

0000000000000000 ltmp0:
       0: 00 14 40 39                   ldrb    w0, [x0, #5]
       4: c0 03 5f d6                   ret

0000000000000008 _bar:
       8: 08 04 c0 39                   ldrsb   w8, [x0, #1]
       c: 09 08 40 39                   ldrb    w9, [x0, #2]
      10: 29 65 1a 53                   lsl     w9, w9, #6
      14: 28 09 48 2a                   orr     w8, w9, w8, lsr #2
      18: 00 1d 00 12                   and     w0, w8, #0xff
      1c: c0 03 5f d6                   ret


[PlainName]

What I don't like in this reoccurring discussion is the mental dishonesty of double stardards.

If you have a nicely designed struct with 57 variables holding your state and you are constructing each of the 57 variables by writing 57 LoC each

Well, actually, the problem we really get is when edge cases are presumed to be the deciding factor. AFAIA, no-one was suggesting using shifts or rotates or whatever to achieve the 3249 lines you suggest would happen¹. Solutions pretty much always depend on the problem to be solved, and in the case of a humongous struct the the solution is likely to be somewhat different to a single 16-bit variable.

Don't forget - the OP asked about a single instance of two-byte data in external storage, and that's what the solutions are (or should have been) aimed at resolving. If you think those solutions would automatically be applied to, say, an ISO 96660 in-memory filesystem then you're just picking obscure cases to have a fight, IMO.

---
¹ OK, there might be a macro that implements such shifts, but then the compiler might do that under the bonnet anyway. Let's not end up working through the assembler then microcode then transistor gates in an attempt to prove something.

[Siwastaja]

Well, actually, the problem we really get is when edge cases are presumed to be the deciding factor. AFAIA, no-one was suggesting using shifts or rotates or whatever to achieve the 3249 lines you suggest would happen¹.
...
Don't forget - the OP asked about a single instance of two-byte data in external storage,

Don't know where you got 3249, maybe by cutting the quote purposefully changing the meaning of the last "each", but no, that was exactly what was suggested, either on purpose or not, but I was completely clear in my earlier post where I started by stating that the ad-hoc arithmetical approach is best for the OP's "one 16-bit variable case", and also that it only breaks down when that comes to 100 LoC constructed the same way, showing that the approach is not generic or scalable; and also that the correct tool depends on the task. Read again, there's nothing unclear in that; I never forgot the OP's case.

Yet exactly all that was shot down by ridiculous straw man arguments ignoring basically everything I wrote.

If people can't handle wall of text, they can choose not to comment on it. Don't do hit-and-run posting looking for a few key words then assuming what is meant instead of reading.

OP's question was already answered and I extended on it and stand by everything I wrote because the big picture is all correct if you take the time to actually read and understand it.

[SiliconWizard]

Just a thought, but next to religion and politics, there's usually nothing more heated than a discussion about programming. It really tends to trigger very strong and opiniated discussions. It just keeps amazing me. Thing is, any topic on programming has a very high probability of ending up in a fight.

I don't think I've ever seen similar things happening with discussions on pure electronics, although I reckon topics on digital electronics also tend to be that way much more than on analog electronics. It may be du to the set of possibilities being so much greater in terms of how to implement one particular feature, or even about methods. Maybe it's also due to concepts being often more complex, and thus, harder to express "right" and easier to misunderstand.

[SiliconWizard]

In this situation I am less concerned about mistyping code. More about having to take care about data types and their ranges. If not that most architectures use two’s complement and compilers use the simplest approach to implementing arithmetic, accidently masking errors in arithmetic, huge portion of C code would explode.

Arithmetic *done right* in C is actually nothing trivial. That's a fact. A number of rules to know and master. Not rocket science, but I admit some can be confusing. And you sometimes have to brush up on them even when you're experienced. That's a very general issue and is of course not a reason for avoiding arithmetic. But that's yet another point in C for which you definitely can't avoid *learning*. As some of us said earlier, for some reason, many people seem to have a problem with actually learning C.

With that said, cases for which you would nitpick using arithmetic or not using it are extremely subjective - that's something, apart from the above point, that you need to realize. And that's probably because of this high level of subjectivity that discussions get very heated.

but I was completely clear in my earlier post where I started by stating that the ad-hoc arithmetical approach is best for the OP's "one 16-bit variable case", and also that it only breaks down when that comes to 100 LoC constructed the same way, showing that the approach is not generic or scalable;

As absolutely any possible construct, except maybe for the most trivial ones. Any form of code duplication is to be avoided at all costs.
If you have to repeat a given conversion/extraction even more than just once, use macros, functions, whatever else available in a given language that gets you to factor this.

With that said, the problem, as I evoked above, is that whatever one considers "trivial" enough not to require factorization is relatively subjective, so even with the things above in mind, we will never avoid strong arguments about a particular approach in a particular case.

[brucehoult]

No comments on the wonders (or evils) of modern C compiler optimisation?

Is it not remarkable to see really quite a lot of code in that macro (could have been an inlined function) reduce down in some cases to just a single machine code instruction?

Constant expression evaluation and dead code elimination do most of the work.

[PlainName]

No comments on the wonders (or evils) of modern C compiler optimisation?

Is it relevant?

Let's say we take a look and don't like what the compiler spits out. Are we going to rewrite the compiler to out preference? Perhaps the output is pretty damn good. Is out C code going to produce the same output with a different compiler? Will we refuse to use compiler A because of the not awfully good optimisation?

Or, horrors of horrors, would we change out C source to persuade the compiler to do a certain thing? If we did that, why the hell are we writing in C and not assembler?

Sometimes optimisation is important, but until then it doesn't really matter what comes out so long as it works. And didn't a certain D Knuth note that premature optimisation is the root of all evil...

(edit: seems that my preferred Dell SK8115 keyboard is throwing a sulk and putting 't' where I really put 'r'. Perhaps it thinks I'm about to replace it.)

[AntiProtonBoy]

I don't get some of those answers. Why to mess with pointers and structures? Uh and memcpy? Seriously?

Assuming this is C and your data are really 16 bit, this is all you need to do:

Yes. Seriourly.

1. Because memcpy is simpler than your shift suggestion:

Code: [Select]

uint16_t original_data;
uint8_t b[2];

memcpy( b, &original_data, sizeof( b ) );

2. In some situations, when strict aliasing rules are in effect, memcpy is the only safe way to transfer multi-byte POD structure and preserve layout.

3. Compilers routinely use memcpy for copying blocks of data to a sequence of bytes, and optimise such operations really well.

[brucehoult]

No comments on the wonders (or evils) of modern C compiler optimisation?

Is it relevant?

Yes.

Let's say we take a look and don't like what the compiler spits out. Are we going to rewrite the compiler to out preference?

I do that from time to time, yes.

Perhaps the output is pretty damn good. Is out C code going to produce the same output with a different compiler?

It's going to produce the same answer with any C compiler, certainly. The exact code generated may be different.

Will we refuse to use compiler A because of the not awfully good optimisation?

Hell, yes!

Or, horrors of horrors, would we change out C source to persuade the compiler to do a certain thing?

Absolutely, in the performance-critical places in our code.

If we did that, why the hell are we writing in C and not assembler?

Because guiding a C compiler is a heck of a lot easier and less error-prone than writing in assembler, and usually gives faster code too -- certainly over a large body of code.

Sometimes optimisation is important, but until then it doesn't really matter what comes out so long as it works. And didn't a certain D Knuth note that premature optimisation is the root of all evil...

Quite apart from speed, code size is often very important, and correctness always is.

Using a generic and tested macro or function is much more likely to give correct code than is writing ad-hoc C code to (in this case) extract aligned and unaligned bitfields of various sizes and locations from a buffer.

[brucehoult]

I don't get some of those answers. Why to mess with pointers and structures? Uh and memcpy? Seriously?

Assuming this is C and your data are really 16 bit, this is all you need to do:

Yes. Seriourly.

1. Because memcpy is simpler than your shift suggestion:

Code: [Select]

uint16_t original_data;
uint8_t b[2];

memcpy( b, &original_data, sizeof( b ) );


I agree with this, but note that it works only if the data is packed to the byte level, not the bit level.

[DiTBho]

sizeof(char) returns 2 on DSP43.
never trust that "char" is 1 byte

[DiTBho]

It is like building cars with wheels that fall off. You know "It doesn't go far, but at least you always know what the problem is!"

that's precisely what happened to me when I build a go-kart driven by a kerosene turbine 
by crashing on the hay bale along the track I learned how to avoid some design mistakes.

[PlainName]

    Is it relevant?

Yes.

Interesting answers, thanks!

I would dispute more than one of them, but I think we are in a different class (for instance, often I don't have a choice of compiler and I don't yet recall ever having the option to write (or rewrite) one).

However, the big one is changing the C source to suit the compiler output. Now, I have done this to circumvent a bug (choice: do it and it works, don't do it and it will crash) but normally I think we consider C to be the lowest level multi-device compatible language. So 'optimising' the C code for the compiler is just making it non-portable and adding in stuff that makes no sense unless one is hip to the why. And doing that when it's not even needed (that is, before you know the code is too slow or whatever) seems quite wrong - it is making it compiler-specific for no practical gain.

[brucehoult]

sizeof(char) returns 2 on DSP43.
never trust that "char" is 1 byte

That absolutely violates the C standard -- sizeof(char) is *defined* to be 1.

Not 1 byte. Just 1. Whatever size a char is, is the unit you measure other things in.

[DiTBho]

That absolutely violates the C standard -- sizeof(char) is *defined* to be 1.
Not 1 byte. Just 1. Whatever size a char is, is the unit you measure other things in.

feel free to send a legal letter to Texas Instruments

[SiliconWizard]

No comments on the wonders (or evils) of modern C compiler optimisation?

To be fair, I did mention the fact modern compilers would often compile either of the approaches suggested in this thread producing almost the same code.
In particular, using bit fields (which is a landmine in terms of portability) with GCC usually yields the exact same assembly code as using bitwise operations. At least I've seen this in tests I had done a couple years ago when I was trying to figure out if one method would be more efficient (regardless of portability or readability) than the other. Turned out the answer was that it made no difference in the general case.