Strategies for ensuring integrity of configuration data

[HwAoRrDk]

I am working on a device that has several hundred bytes of configuration data. Permanent storage of this data is in EEPROM, but is maintained in RAM during runtime. However, I am worried about preserving the integrity of this data. The reason why is that my device will potentially have power-on periods of weeks to several months. Most of the power-on time will be spent in a standby/sleep state.

What strategies can people recommend for ensuring the integrity of the configuration data in RAM during such long operation periods?

At the most basic level, I had the idea to keep a CRC checksum of the entire config struct, and to periodically recalculate it and verify that both match. If not, re-read the data from EEPROM.

I understand there are different CRC types intended for different purposes (e.g. designed to catch transmission errors). Would there be one most suited to my scenario?

What about CRC length? I am inclined to go with 16 bits, as avr-libc already has an implementation for that.

[rhb]

How fast does it need to start up?  Can you reload RAM from EEPROM when it wakes up?  Otherwise sleeping on a timer interrupt that either checks the CRC of RAM or compares the RAM to the EEPROM at regular intervals should do a good job of making sure the RAM contents are correct.  As it is only a few hundred bytes, either approach should be very quick and consume very little power.

[cv007]

Check the ram retention voltage in your datasheet. It would be odd if not below brown-out reset voltage (possibly quite a bit below).
If you doubt your ram is being retained 100% in sleep, its not going to do much good to just check the config data in ram, now you will have to check everything in ram and at that point you may as well reset on every wake-up.

If you have power above ram retention voltage, your ram remains as it was. If you go below that voltage, the brown-out reset already tripped long ago. If you don't use brown-out reset, then the por arm/rearm voltages may need to be checked in the datasheet against the ram retention voltage. You probably then want to reset after a wakeup if the possibility exists to get close to the ram retention voltage without a por.

[SiliconWizard]

Checking a CRC is a good idea. You could also keep a CRC in EEPROM while you're at it, so you can check integrity of data in EEPROM while you read it from there. EEPROM data corruption can also happen. So keeping two identical blocks of configuration data in EEPROM (ideally in completely different "pages") could be a good idea too, so you have a fallback if one of them gets corrupted.

Then what to do next all depends on how fast the access to configuration data needs to be, as rhb said.
Strategies can go from reading configuration data from EEPROM every time you need it, keeping a RAM shadow of it as you suggested and update it (re-read from EEPROM) when the CRC is wrong, updating the RAM shadow at regular intervals no matter what (if you can afford it), etc. Obviously if there is a real-time concern, you also have to figure out what to do in case of corruption due to the overhead of re-reading from EEPROM, like using hard-coded default parameters while the EEPROM is getting read.

If RAM shadowing is a necessity due to performance reasons, an additional strategy can be to keep two RAM shadows of the EEPROM data (each with its CRC). Then you can read from either (in an alternate way for instance). If the CRC of the current shadow is wrong, then you read from the other. If both CRCs are wrong, you update both shadows from EEPROM. This would lower the probability of having to read from EEPROM, which can be an advantage for performance or power consumption reasons.

The kind of CRC to use is all a question of probability, so there is no "right" answer here IMO. In many cases, for data storage, even a simple 8-bit checksum is enough to provide reasonable integrity check as long as the probability (and extent) of data corruption is low and the data size is small. Of course if your device could be subjected to radiation, the probability of RAM corruption would increase dramatically.

In many cases, I now tend to use a CRC-32, but that's pretty much a matter of compromise between robustness and performance on the targets I deal with most of the time.

[Jeroen3]

Before you start working on a problem that does not exist.
What happens if the data gets corrupted?
Does it stop working? Does it damage something? Is it dangerous for people?
What is the probability of read errors?

You can buy a chip that has a checksum on the SRAM, you can also buy chips with correcting memory.
The chances of corruption are then lowered, not removed. But at least you'd know, most of the times.

[HwAoRrDk]

How fast does it need to start up?  Can you reload RAM from EEPROM when it wakes up?  Otherwise sleeping on a timer interrupt that either checks the CRC of RAM or compares the RAM to the EEPROM at regular intervals should do a good job of making sure the RAM contents are correct.  As it is only a few hundred bytes, either approach should be very quick and consume very little power.

It doesn't need to start up particularly fast - even a few hundred milliseconds would not have any effect on anything - so I suppose I could reload from EEPROM. Yes, I may do that.

In the power-down mode I'm using, the only timer that can cause wake-up is the watchdog timer, and that has a maximum period of 8 seconds; that seems far, far too frequent for that kind of thing, so I'm not sure I'd want to do that. I suppose I would need to measure the current draw during a wake-up-and-check event to see if it was acceptable in terms of power usage.

Check the ram retention voltage in your datasheet. It would be odd if not below brown-out reset voltage (possibly quite a bit below).
If you doubt your ram is being retained 100% in sleep, its not going to do much good to just check the config data in ram, now you will have to check everything in ram and at that point you may as well reset on every wake-up.

If you have power above ram retention voltage, your ram remains as it was. If you go below that voltage, the brown-out reset already tripped long ago. If you don't use brown-out reset, then the por arm/rearm voltages may need to be checked in the datasheet against the ram retention voltage. You probably then want to reset after a wakeup if the possibility exists to get close to the ram retention voltage without a por.

I checked the datasheet, and it doesn't give a RAM retention voltage. But it does specify a minimum voltage to sustain power-down sleep mode, which is 2.5V. I'm running the device at 5V, and I have the brown-out detection enabled and set to 4.3V, so corrupting memory due to voltage drop isn't really a concern, as the device will reset in those situations.

Checking a CRC is a good idea. You could also keep a CRC in EEPROM while you're at it, so you can check integrity of data in EEPROM while you read it from there. EEPROM data corruption can also happen. So keeping two identical blocks of configuration data in EEPROM (ideally in completely different "pages") could be a good idea too, so you have a fallback if one of them gets corrupted.

Good point, yes, I could also keep a CRC of the EEPROM contents too. Wouldn't hurt to apply belt and braces.

If RAM shadowing is a necessity due to performance reasons, an additional strategy can be to keep two RAM shadows of the EEPROM data (each with its CRC). Then you can read from either (in an alternate way for instance). If the CRC of the current shadow is wrong, then you read from the other. If both CRCs are wrong, you update both shadows from EEPROM. This would lower the probability of having to read from EEPROM, which can be an advantage for performance or power consumption reasons.

Yes, RAM shadowing is fairly important for performance reasons. Part of the configuration data is several 'programs' in bytecode for a stack-based VM that get executed hundreds of times per second, so delays in accessing that data are particularly undesirable.

Not sure whether I may have the spare RAM to keep two copies of the configuration data. Besides that, I'm not sure how I would go about abstracting the swapping between the two copies. I suppose it could be as simple as changing a pointer to a struct. Perhaps have a timer interrupt (maybe the watchdog, as mentioned above) do the CRC check on the secondary copy, re-load from EEPROM upon failure, then if all okay swap pointers. I wouldn't want to do any verification and re-loading on the active copy, as that would probably delay normal functions too much.

Before you start working on a problem that does not exist.
What happens if the data gets corrupted?
Does it stop working? Does it damage something? Is it dangerous for people?
What is the probability of read errors?

Corruption could lead to totally incorrect behaviour, which at worst would mean the device becomes useless. Would that lead to damage or danger? No. But it sure would piss off the user.

I of course can't say with any certainty what the probabilities of failure might be. Just that the thought occurred to me that when dealing with run times potentially extending into the months, well, entropy happens, so...

[cv007]

I'm running the device at 5V, and I have the brown-out detection enabled and set to 4.3V, so corrupting memory due to voltage drop isn't really a concern, as the device will reset in those situations.

Then what else is going to corrupt the memory? And if you think something is going to corrupt it, why will it only happen to the config data in ram and not also all the other ram, along with your cpu registers? How about the program counter?

Go find all the manufacturers app notes, forum posts, etc., on what you want to do. There is a reason you won't find anything. I would take that as a hint that the problem does not exist. Doesn't hurt to think about what it takes to retain ram memory, though.

[Siwastaja]

SRAM is typically considered "perfect", unless it's a super safety critical device, in which case you want to have HW solutions such as parity checking, running multiple devices in sync to vote, etc., which all are very complex to prove to improve safety at all.

In that case, only checking errors in a subset of the memory space - only the configuration parameters - makes very little sense, as corruption anywhere else would cause at least similar effects - most likely even worse, since configuration parameters are typically trivial to sanity-check, while corruption on stack variables, etc., would be something you normally won't check while running. The fact the data is unchanged for a long time does not matter - single corruption anywhere could wreak havoc, regardless of whether the data was written 1 us or 1 year before.

Most likely, the thing is: no, it makes little sense to add safeguards on the SRAM configuration parameter section. By luck, it could detect a programming error, or a rare glitch, but it's unlikely the glitch would hit such a small memory section (some percents of the full memory range?)

(By all means add checksumming to the EEPROM section, however, and check it at the load).

If this does matter (being an extra critical application), what you suggest certainly isn't sufficient at all, and is a totally wrong solution offering minor, partial, almost negligible protection, while you need a more generic protection scheme for your potential SRAM corruption risk. But I doubt this is the case; if it was, you wouldn't be asking it like this.

[HwAoRrDk]

Sounds like I'm worrying needlessly. Okay, I will forget about verifying the configuration data in RAM. But I think I will still put a CRC on the EEPROM data.

I will have to decide what exactly to actually do if the CRC check fails when loading from EEPROM, though, as there aren't really any sensibly useful default values I can use (because the whole operation of the device hinges almost entirely on the user's configured parameters), nor any particularly noticeable way of alerting the user to this scenario, apart from the device doing essentially nothing. I'm sure I'll figure something out.

Thank you all.

[Jeroen3]

...
Go find all the manufacturers app notes, forum posts, etc., on what you want to do. There is a reason you won't find anything. I would take that as a hint that the problem does not exist. Doesn't hurt to think about what it takes to retain ram memory, though.

The problem does not exist on the surface of earth.

[rhb]

First of all cosmic rays and background radiation (e.g. proximity to the Chattanooga shale)  can change the state of a single bit in memory at *any* time.  So the OP's concern is legitimate.

I have encountered two instances where DRAM decayed over the course of a week.  So an operation that worked immediately after booting would cause a kernel panic a week later.  I have tested this rigorously by booting the system, running the command, leaving it idle for a week and rerunning the command.  And this is on workstations with ECC memory.  I currently have a bad DIMM is my Solaris 10 system.  If I run a zpool scrub more than a few days after I boot the system it will kernel panic.  Because accessing memory refreshes it, normal memory tests will not find the bad DIMM.  I'm still trying to figure out how to locate the bad DIMM.

Aside from keeping two copies of the constants and the CRC, you could use a multibit ECC that was able to recover from bit errors in say 1/2 of the constant block.  There's plenty of literature on the subject, but if you need a reference I probably can find one.  It's likely to be old, so look on line first.

[NorthGuy]

The problem does not exist on the surface of earth.

The problem certainly exists somewhere on the Earth. Otherwise you wouldn't see ECC memory. Moreover, some vendors go great lengths to address the problem. For example, Xilinx stores Humming codes for every frame of the configuration memory of their FPGA, and also has hardware which constantly check the codes against the real contents and can notify the user's design if memory changes.

[nctnico]

SRAM is typically considered "perfect", unless it's a super safety critical device, in which case you want to have HW solutions such as parity checking, running multiple devices in sync to vote, etc., which all are very complex to prove to improve safety at all.

Well, don't rule out an event where a roque pointer writes data where you don't expect it.

[wraper]

There is no a good reason why SRAM content would go bad. Just think about something like Dallas DS2250 encrypted microcontroller modules. They have no flash or eeprom at all, SRAM only. And I can say from experience, they still run after 2 decades since firmware was loaded into them.

[wraper]

The problem does not exist on the surface of earth.

The problem certainly exists somewhere on the Earth. Otherwise you wouldn't see ECC memory. Moreover, some vendors go great lengths to address the problem. For example, Xilinx stores Humming codes for every frame of the configuration memory of their FPGA, and also has hardware which constantly check the codes against the real contents and can notify the user's design if memory changes.

Don't confuse SRAM and DRAM. I'd be more concerned about reliability of EEPROM/FLASH contents, as long as power is stable and there is no strong ionizing radiation. But even then it probably would sill last better.  It's not like a single particle could cause a bit flip in low density SRAM.

[Siwastaja]

SRAM is typically considered "perfect", unless it's a super safety critical device, in which case you want to have HW solutions such as parity checking, running multiple devices in sync to vote, etc., which all are very complex to prove to improve safety at all.

Well, don't rule out an event where a roque pointer writes data where you don't expect it.

Which I explicitly mentioned: search for "programming error" in my post.

I mentioned it for a good reason; it is actually about a million times more probable than an actual SRAM HW bit error, especially for a beginner or someone not working under very strict control and tooling.

But, to better catch this failure mode as well, you'd want to guard as large area as possible, not only a minuscule configuration parameter section. But this quickly becomes non-trivial.

[NorthGuy]

And I can say from experience, they still run after 2 decades since firmware was loaded into them.

This doesn't prove anything. I have lived for much longer than 2 decades and I hasn't been killed in a traffic accident. Does this mean humans never die in traffic accidents?

[Siwastaja]

Do note that because SRAM is similar in construction to logic cells on the same CMOS fabric, kind of similar chances of flipping a bit also exist in the CPU/peripheral registers (internal and accessible) as well as internal data and control paths, and combinatorial logic. Heck, partially this is even well known and analyzable; IO and clock domain synchronization double or triple flip-flops have a certain, fairly well modeled and known chance of hitting metastability, which can propagate in the system and cause unexpected behavior; this doesn't even require cosmic rays or power problems. It's a certain MTBF.

Microcontrollers often have such small SRAM that it isn't even statistically much bigger target than the CPU core, for example.

This risk is very small, but it's similar to the risk of the SRAM corruption.

This is why both super-high-reliability products, and coding paradigms are very demanding: you have to protect everything. This is expensive, slow to learn and slow to implement, requiring massive budget and really competent engineers. I have a good idea what it is like, and I know I couldn't do that job - even if I could get years of extra training. This is relevant in some small subset of automotive, aviation and mass destruction technology; basically, where a bitflip could directly kill large masses of people.

In consumer and 99.9% of industrial technology - including most of the industrial robotics capable of killing in bad conditions - this is something that is completely ignored, simply assuming no hardware corruption of embedded SRAM memory or CPU logic. Everything does have a MTBF, but it's so high it completely gets drowned in noise, because incidents due to orders of magnitude more common causes happen.

Reliability optimization is like any optimization, and at least understanding the concept of Amdahl's law (but for MTBF instead of execution speed) is highly relevant.

But do read about ways to write more robust microcontroller software, because you will eventually write a bug which corrupts the memory, or does something else similarly disturbing and hard to find.

[Jeroen3]

How do you check if the flash is still correct? This is something I'd be more concerned about.
(I check for that in my projects)

[Siwastaja]

Yes, even many non-safety, normal-reliability products (such as STM32H7) do have parity generation and checking of the flash.

This is easier to protect against in most applications, since you don't write it, so you can store a compile-time-generated CRC and do periodic full flash reads and CRC checks and hope to catch the corruption before it does something unwanted, like run a corrupted instruction. But it could still happen between such checks; you'd need the actual parity check HW that works on every flash read operation and instantly gives an interrupt to catch that.