Author Topic: memset() and \0 termination  (Read 5227 times)

0 Members and 1 Guest are viewing this topic.

Offline ricko_ukTopic starter

  • Super Contributor
  • ***
  • Posts: 1233
  • Country: fr
memset() and \0 termination
« on: June 30, 2021, 02:10:18 am »
Hi,
couple of questions:

1) if you do not initialise a char[], does the compiler automatically fill it with '\0'? Are there variations depending on the compiler standard/version?

2) the following code snippets should yield the same result and NOT go out of boundaries or anything else correct? Asking because the debugger sometimes seems to show different behavious in the last char

#define SIZE 4
char sz_dest[SIZE];
memset(sz_dest, 'A', SIZE);
sz_dest[SIZE - 1) = '\0"

and

#define SIZE 4
char sz_dest[SIZE];
memset(sz_dest, 'A', SIZE-1);
sz_dest[SIZE - 1) = '\0"

Thank you
 

Offline Fredderic

  • Regular Contributor
  • *
  • Posts: 69
  • Country: au
Re: memset() and \0 termination
« Reply #1 on: June 30, 2021, 02:23:42 am »
I'm no expert…  So just filling in until someone better comes along.  But…

  • "It depends".  If your char is global, then the memory it occupies will probably be zero-initialised.  If it's stack or heap allocated, then definitely not.
  • Typo's aside, they should have the same result…  the last char being determined by the last statement, which is the same in both cases.  The first one would just set the last char twice.
 

Offline ricko_ukTopic starter

  • Super Contributor
  • ***
  • Posts: 1233
  • Country: fr
Re: memset() and \0 termination
« Reply #2 on: June 30, 2021, 02:57:02 am »
Thank you Fredderic :)

1) yes, I assumed so but was wondering if there are differences between compiler standards
2) found what was causing the strange behaviour

Thank you
 

Offline ejeffrey

  • Super Contributor
  • ***
  • Posts: 4865
  • Country: us
Re: memset() and \0 termination
« Reply #3 on: June 30, 2021, 04:11:24 am »
Initialization of global / static variables to zero is part of the C standard.  That doesn't mean you can't muck around the the linker script to make it non compliant -- but initialization is standard and all compilers should do this in their normal configuration.

Automatic variables and malloc() allocations don't do initialization but you have to watch out.  The OS may zero-initialize memory before giving it to the application process.  This can lead to people believing that their compiler is doing initialization until you do a few malloc/free cycles and are now reusing heap memory.  calloc() does zero-initialize the memory.
 

Offline ricko_ukTopic starter

  • Super Contributor
  • ***
  • Posts: 1233
  • Country: fr
Re: memset() and \0 termination
« Reply #4 on: June 30, 2021, 04:52:32 am »
Thank you Ejeffrey :)

yes I thought the linker might affect that but wasn't sure because I never played/modified linkers before.
 

Offline Siwastaja

  • Super Contributor
  • ***
  • Posts: 11442
  • Country: fi
Re: memset() and \0 termination
« Reply #5 on: June 30, 2021, 06:15:23 am »
If your char is global, then the memory it occupies will probably be zero-initialised.

Not probably, it is. Doesn't need to be global, static inside a function is zero-initialized too.

Exception to this rule would be a custom startup code intentionally breaking the C standard. You know if this is the case (basically, you have written the custom startup code.)

Because this is basic 101 C and so widely trusted, I would be surprised to see buggy implementations out there. You can trust it. If it doesn't work, then ditch the niche compiler in question, it's likely completely unusable in other ways, too.

The next question would be, when such "uninitialized" global/static are initialized to zero? The answer is, before main() by the startup code compiler inserts. Sometimes on MCUs you need extremely short startup times which could be a reason to write your own startup code, for example performing something important before the .data copy and .bss zeroing.

Local variables need explicit initialization. Beware, contents can be anything that happens to be in that part of memory from previous operations, this is a classic security leak, maybe you happened to store some secret nuke launch code in the same memory area before!

memset is a classic for initializing local (stack) variables. To reduce risk of bugs when modifying the code later, usage of sizeof is recommended in the last argument.

What comes to the code examples, functionally they do the same. The former just does unnecessary work by first memseting the last element to 'A', then overwriting it to 0.

But please, copy-paste the same code which you have compiled and tested, this way you ensure there are no new typos included.
« Last Edit: June 30, 2021, 06:23:37 am by Siwastaja »
 
The following users thanked this post: peter-h, newbrain

Offline ajb

  • Super Contributor
  • ***
  • Posts: 2867
  • Country: us
Re: memset() and \0 termination
« Reply #6 on: June 30, 2021, 03:08:35 pm »
What Siwastaja said.  There is also a shorthand for zero initialization, which is to do:

Code: [Select]
char someArray[SIZE] = {0};

In C, if an array is partially initialized, any extra elements that aren't covered by the initialization list will be initialized to zero. So you only need to be explicit about the first element and everything else will be zeroed implicitly.

That's all about what the compiler will do for you, and anyone can reference your code against the C standard to determine what will or will not be initialized to zero, however this will not tell them what you WANTED to be initialized to zero versus what's getting zeroed just because that's the compiler's default behavior.  So if you have an array that MUST be initialized to zero for the rest of the code to function properly, then it's a good idea to document that, which you can do using the zero initialization above as a hint that the zero initialization is deliberate--of course you can also just use a comment.  That said, if your code depends on that zero initialization then you may want to include zeroing the array in your startup or initialization code.  The implicit zero initialization may be adequate when the application starts up, but sometimes you need to be able to reset/reinitialize a module within your application, and if that module requires an array that starts out zeroed you will need to reinitialize it explicitly.  This is one aspect of writing maintainable, well-decoupled software: each module of an application should be capable of independent and complete initialization, without depending on the rest of the application being in any particular state or providing any additional setup.  That's the ideal anyway, of course in reality that's not always the way things happen for many good (and bad) reasons, but at the very least it's worth keeping in mind as you start laying out a software project.
 

Offline SiliconWizard

  • Super Contributor
  • ***
  • Posts: 17840
  • Country: fr
Re: memset() and \0 termination
« Reply #7 on: June 30, 2021, 05:46:18 pm »
1) if you do not initialise a char[], does the compiler automatically fill it with '\0'? Are there variations depending on the compiler standard/version?

As others have already said, it is guaranteed by the C standard, as long as said array is a global or static local array.
OTOH, a non-static-qualified array local to a function is NOT automatically intialized with all zeroes. Some specific compiler options may allow this, but this isn't guaranteed by the standard, and would be compiler and option-specific. So, always initialize your non-static local arrays if they need to be.

2) the following code snippets should yield the same result and NOT go out of boundaries or anything else correct? Asking because the debugger sometimes seems to show different behavious in the last char

#define SIZE 4
char sz_dest[SIZE];
memset(sz_dest, 'A', SIZE);
sz_dest[SIZE - 1) = '\0"

and

#define SIZE 4
char sz_dest[SIZE];
memset(sz_dest, 'A', SIZE-1);
sz_dest[SIZE - 1) = '\0"

Both of your examples are fine.
Of course, a couple points:
- The sz_dest[SIZE - 1) = '\0'; part is unnecessary if again the array is a global array. If it is local, then it is necessary.
- The same thing could be just written: char sz_dest[SIZE] = "AAA";
Your version has the benefit of being able to accomodate any length by just modifying the SIZE macro. But, unless you have a specific reason for specifying the size with a macro this way, note that a much simpler version would just be:

char sz_dest[] = "AAA";

This syntax is valid in gloabl as well as local contexts. The size of the array is determined by the constant at compile time, and in this case, it will be 4, as string literals always include the final zero in C.

 

Offline westfw

  • Super Contributor
  • ***
  • Posts: 4670
  • Country: us
Re: memset() and \0 termination
« Reply #8 on: July 01, 2021, 08:14:45 am »
Quote
Initialization of global / static variables to zero is part of the C standard.
We had a system that initialized RAM to 0xFF (heap for sure.  BSS as well, I think.)  Because if you started followed wild pointer chains, 0 would be a valid pointer (in the sense that there was real memory there), but -1 definitely wasn't.  I suspect that that was undone a long time ago, though.  "cleverness" is one thing, but once you reach a certain size, "doing what every expects" becomes a lot more important.
 

Offline harerod

  • Frequent Contributor
  • **
  • Posts: 553
  • Country: de
  • ee - digital & analog
    • My services:
Re: memset() and \0 termination
« Reply #9 on: July 01, 2021, 04:12:02 pm »
Quote
Initialization of global / static variables to zero is part of the C standard.
We had a system that initialized RAM to 0xFF (heap for sure.  BSS as well, I think.)  Because if you started followed wild pointer chains, 0 would be a valid pointer (in the sense that there was real memory there), but -1 definitely wasn't.  I suspect that that was undone a long time ago, though.  "cleverness" is one thing, but once you reach a certain size, "doing what every expects" becomes a lot more important.

Initialising RAM to a non-zero pattern may help to figure out memory usage. Its usefulness depends on the actual application. It can help with questions like:

- How far are stack and heap apart?  (non-os platforms)
- Which portions of memory have been written? (wild pointers)



 

Offline SiliconWizard

  • Super Contributor
  • ***
  • Posts: 17840
  • Country: fr
Re: memset() and \0 termination
« Reply #10 on: July 01, 2021, 04:48:53 pm »
Quote
Initialization of global / static variables to zero is part of the C standard.
We had a system that initialized RAM to 0xFF (heap for sure.  BSS as well, I think.)

As others have said above, this just isn't a standard-compliant implementation if it does that with the .bss segment.

But, certainly, some tools may do stuff that are not compliant. The important point is knowing your tools, and in particular knowing what it is they do that is not compliant.

Also, as someone mentioned, this stuff can be, and is usually done in the "startup" code. In particular, zero-ing out the .bss segment is done in it. It's pretty easy to modify it and initialize this segment with other values. It's just non-compliant.

As to the heap or stack, the initial content is unspecified in the standard AFAIK, so you're free to fill them with whatever you please. That would, again, usually be done in the startup code. Although most implementations just leave them as they are, some may indeed fill them with specific "magic" values. For instance to analyze usage, as was talked about in the thread in which we discussed stacks.

Just a quick additional note: even though the standard ensures zero-initiliazed global/static variables (when compliant), I for one have never relied on this. if anything, not necessarily because I do not trust my tools to be compliant, but because I think it makes code less readable. So I usually always initialize them (when they need to be, otherwise I don't care) with some value. Even if it's just 0.

Edit2: come to think of it, filling the .bss segment with something else than zero in the startup code would not work if the compiler itself is compliant. This is why: global/static variables that are explicitely initialized *with zero* will also end up in the .bss segment. At least that's what GCC does. So if the startup code fills the segment with anything else than 0, that would break the code badly. Unless you make it a rule NEVER to explicitely initialize those variables. That would be extremely confusing though. So I doubt your tools were doing this with the .bss segment. But, as I said above, that would be fine for the heap and the stack.
« Last Edit: July 01, 2021, 05:10:28 pm by SiliconWizard »
 

Offline Siwastaja

  • Super Contributor
  • ***
  • Posts: 11442
  • Country: fi
Re: memset() and \0 termination
« Reply #11 on: July 01, 2021, 06:07:41 pm »
This is why: global/static variables that are explicitely initialized *with zero* will also end up in the .bss segment. At least that's what GCC does.

... or not. I mean, it's a sane thing for a compiler to do, but in my experience, this doesn't necessarily happen.

One of the reasons I picked up a habit of not explicitly initializing globals/statics to zero was that exactly GCC (forgot which version, which platform - possibly 4.something on AVR or ARM) put explicitly zero-initialized in .data, consuming flash memory by storing all the zeroes. I also picked up the habit of commenting it "initialized to zero by C standard" but dropped it at some point assuming this is such basic feature in C anyone who works with my code should just know this, like they are supposed to know how the ?: operator works, and that bitshift operators for signed integers are arithmetic and not logical shifts, for example; those are not obvious to beginners, either, and no one wastes a comment how they work unless it's a tutorial. Comments are better used to describe design decisions ("why" and bigger-picture "how"), and "what" only when it's some really weird / rarely known.

Indeed, the whole reason I can think of why .bss exists in the first place is optimization of program size because zero is the single most usual initialization value. Otherwise than that, zero-initialization of globals/statics could be fulfilled by automatically adding them to .data with zero values, by compiler. While arbitrary initialization is expensive, if startup time is of no consideration, initializing to zero is a very low cost thing to do, to the point of C standard writers not coming up with an alternative way to tell the compiler not to initialize globals/statics. Initializing stack objects to zero would insert the delays not to startup but all over the program so clearly they thought programmer needs to do that explicitly, which was not a bad idea.
« Last Edit: July 01, 2021, 06:12:36 pm by Siwastaja »
 

Offline SiliconWizard

  • Super Contributor
  • ***
  • Posts: 17840
  • Country: fr
Re: memset() and \0 termination
« Reply #12 on: July 01, 2021, 06:14:22 pm »
This is why: global/static variables that are explicitely initialized *with zero* will also end up in the .bss segment. At least that's what GCC does.

... or not. I mean, it's a sane thing for a compiler to do, but in my experience, this doesn't necessarily happen.

One of the reasons I picked up a habit of not explicitly initializing globals/statics to zero was that exactly GCC (forgot which version, which platform - possibly 4.something on AVR or ARM) put explicitly zero-initialized in .data, consuming flash memory by storing all the zeroes.

I haven't seen that in GCC for many versions, but I admit I absolutely don't remember the exact behavior of GCC 4.x or older. It doesn't matter.

My point was: it's reasonable for a compiler to do it, and as it is allowed to do so from the standard, any compiler *may* do this, and GCC has certainly been doing this for a number of versions now. Which is a good reason enough not to initiliaze the .bss segment in the startup code with anything else than 0. Otherwise you're bound to break code in very nasty ways. End of the story. That was just a rationale for never fiddling with the .bss segment in this way.

And that said, whereas we can see a benefit of initializing the heap and the stack with magic values (and other than zero), I can't see any for the .bss segment. So point was, it's extremely unlikely any platform would have done this.

And the main point of the .bss segment is indeed to save code size. Storing unnecessary zero values in code is a waste of code memory. It can make a lot of difference if you have big global arrays and data structures.

« Last Edit: July 01, 2021, 06:17:06 pm by SiliconWizard »
 

Offline Siwastaja

  • Super Contributor
  • ***
  • Posts: 11442
  • Country: fi
Re: memset() and \0 termination
« Reply #13 on: July 01, 2021, 06:17:54 pm »
Indeed, if you want to leave something uninitialized, or initialize to other weird values than zero, don't mess up with .bss init code, instead add another, clearly named section like .noinit or .init_to_42 and do whatever you want with it in init code. This way you have the control, and still retain standard-compliant operation on everything which do not carry the very visible __attributes__ in code.
 
The following users thanked this post: newbrain

Offline SiliconWizard

  • Super Contributor
  • ***
  • Posts: 17840
  • Country: fr
Re: memset() and \0 termination
« Reply #14 on: July 01, 2021, 06:21:13 pm »
Sure, but I still can't see any reason for doing this. Absolutely none.
If you're doing this just to keep track of unitialized global variables, any decent static analysis tool will tell you that without requiring to get to the link phase and mess with segments.
If you're doing this to automatically initiliaze global variables with a value other than zero, well, as I said earlier, I think it would be an atrocious decision from any readabilitu and code maintenance POV. Just never even think of that. Even the thought of it sounds twisted. >:D
 

Offline Siwastaja

  • Super Contributor
  • ***
  • Posts: 11442
  • Country: fi
Re: memset() and \0 termination
« Reply #15 on: July 01, 2021, 06:56:18 pm »
One reason: Startup time. Often extremely important in embedded. Clearing .bss takes time. Not much, but measurable in microseconds, maybe even some single-digit milliseconds!

Although, I prefer to add the Critical Things in the startup code before .bss zeroing / .data copy.

But sometimes you don't have a few Critical Things but instead, want to run the whole thing i.e. enter main() as soon as possible.

Especially worth noting is modern MCUs boot with some super slow internal RC oscillator. You may want to switch to an external clock sooner than at main(). ... and do .bss/.data loops while waiting for the PLL stabilization flag  :-/O

You may also want to retain values through software reset for any weird reason.
« Last Edit: July 01, 2021, 07:00:24 pm by Siwastaja »
 

Offline SiliconWizard

  • Super Contributor
  • ***
  • Posts: 17840
  • Country: fr
Re: memset() and \0 termination
« Reply #16 on: July 01, 2021, 07:17:18 pm »
One reason: Startup time. Often extremely important in embedded. Clearing .bss takes time. Not much, but measurable in microseconds, maybe even some single-digit milliseconds!

But. I'm not sure I get what you really meant then. I was more talking about automatically initializing global variables with a value other than zero. Here you are talking about not initiliazing them at all. That's  different. And, in this particular case, yes, I can see the benefit (as long as you are cautious when writing code).

Actually, possibly the most efficient approach would be:
- Put global initializers in the .data segment as usual (those that are not all zeroes);
- Put non-initialized global variables in the .bss segment;
- Except those that you explicitely locate (with an attribute) in a separate dedicated segment.

Now that you explained your rationale, that's probably exactly what you meant. And then, I agree.

Especially worth noting is modern MCUs boot with some super slow internal RC oscillator. You may want to switch to an external clock sooner than at main(). ... and do .bss/.data loops while waiting for the PLL stabilization flag  :-/O
Well yeah. On some MCUs, the "boot" time can be pretty long, and variable initialization at startup is actually pretty negligible in most cases. In some particular cases, as you mentioned, you may optimize things with the PLL, if using any. That would require putting the PLL config code in the startup code, which is OK but probably less "readable" than having it in your source files. But, for some MCUs which have no embedded Flash and run code from external Flash, really, the boot phase usually exceeds by a large margin anything you do in the startup code... so yeah.

You may also want to retain values through software reset for any weird reason.

That would be a good reason, as long as said processor guarantees RAM integrity after a reset. On some, there are a number of dedicated registers instead. Or, a small RAM area dedicated to this. Of course, then, you need to define it in the linker script as a dedicated segment.

Which leads us a bit astray, because I was of course not questioning the potential benefits of defining custom segments.
 

Offline bson

  • Supporter
  • ****
  • Posts: 2822
  • Country: us
Re: memset() and \0 termination
« Reply #17 on: July 01, 2021, 10:11:43 pm »
There are very good practical reasons for retaining memory across reset.  I sometimes put aside a small data section in memory as a ring buffer, and define putc() or the equivalent delegate to stick trace messages there in a debug build.  Then if it resets the first thing I do is if there's something in the ring stop and wait to attach a debugger with a flashing LED.  (Same if the debug build panics.)  I attach a debugger and can check what the last 2k or so of console output was to see what it was doing during the reset.  This is very useful for debugging things like spurious watchdog and brownout resets.  (And, of course, if the processor has a register indicating why it reset, I stash that away as well at startup.)  For example, if you go to switch a relay and the relay switches and but the processor resets (which may be so fast you don't even notice it if you aren't looking for it) it's probably due to a supply drop.
 

Offline westfw

  • Super Contributor
  • ***
  • Posts: 4670
  • Country: us
Re: memset() and \0 termination
« Reply #18 on: July 02, 2021, 12:12:07 am »
Quote
If you're doing this just to keep track of unitialized global variables, any decent static analysis tool will tell you that
Kids today!The aforementioned hack was from back in the mid-80s, before gcc existed and probably before "decent static analysis" tools existed, either.  (and it might have been back when the C specification was "weaker" as well; certainly before compiler "compliance" was a big issue.)
 

Offline Siwastaja

  • Super Contributor
  • ***
  • Posts: 11442
  • Country: fi
Re: memset() and \0 termination
« Reply #19 on: July 02, 2021, 02:05:28 pm »
But. I'm not sure I get what you really meant then. I was more talking about automatically initializing global variables with a value other than zero. Here you are talking about not initiliazing them at all. That's  different.

Scroll up.

Me: "if you want to leave something uninitialized ..."
You: "Sure, but I still can't see any reason for doing this. Absolutely none."

Replied to that.
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf

 

- - -
-->