General > General Technical Chat
Preserving settings over a firmware upgrade
(1/8) > >>
AndyC_772:
Hi,

I have a product which has been in production for a number of years, and which has a lot of configurable settings. Some of these are set at the factory and are hidden, while others are accessible to the end user.

The environment is an ARM microcontroller using internal storage only. No file system or networking, but it can be connected to a PC via a serial link.

Typically I store all of a product's settings in a C struct called 'nv' ('non-volatile'), which is stored in Flash whenever a change is made, and read back in as part of the boot process.

Within this struct there are all kinds of integers, strings, other structs and so on, and then right at the very end there's a uint32_t which must always be set to a 'magic number' to indicate that the rest of the struct has been populated with valid data. This number could be a fixed value, a CRC, a cryptographic hash, checksum, or whatever.

When the firmware boots, it checks for the magic number. If it's valid, then that's great, carry on as normal. If not, it assumes the configuration is broken, and restores factory default settings including a new magic number.

This works well. It ensures that the unit always boots with valid settings whether it's brand new and is booting for the first time, or if it's been updated to new firmware which (usually) has a larger 'nv' struct with the magic number in a different place.

Until quite recently, firmware updates have only ever been done at a very limited number of authorised sites, where having the unit reset to factory settings after an update isn't a problem. However, I'd now like to explore how to go about preserving settings when a unit is updated by the end user in the field.

The issue is, of course, that new firmware may well require some new settings that weren't supported before. Insert new items into a struct, and everything beyond them has to move.

A few options spring to mind, but I can't quite figure out which I dislike least.


* Keep the original nv struct exactly the same forever. For new settings introduced in the next firmware release, create a new struct called nv2. Then nv3, nv4 etc, each with their own magic number to show they're valid.
* As above, but add the new items to the end of nv so they sit after the magic number, which at least means I don't have to litter my code with nv3.this, nv5.that, nv2.the_other etc
* Reserve space in nv between the last used setting and the magic number, put any new settings here. The magic number doesn't move, so it definitely has to be a CRC or hash, not a fixed value.
* Tag the nv struct with a value that indicates its version number. Include start-up code that recognises previous versions and knows how to migrate settings to the current version.
* Export settings in a readable format, with tags or commands to indicate what each value actually means. After an upgrade, reset to factory defaults, then parse the readable version to get back to a binary.
* The better idea that I haven't thought of yet...!
Suggestions please, folks. What's the industry's solution to this problem?
tszaboo:
Think of it from the hardware perspective for a second. You can only erase sectors, that are for example 4KB in size, and you need to erase them to overwrite data in it. Is 4KB enough for you, than make the last 4 bytes eg. for your CRC and magic number, and place the data before, always.
How you organize your data before it... there are so many ways to do that I'm sure you find something you like. I think last time I was doing something like this, I had a lightweight version control for the config file, and the config version was the first data saved.
Veteran68:
Several ways to go about this, but just to clarify: are you considered concerned about the storage space available and ensuring it's contiguous, or are you simply concerned about the management of the code itself and how the struct is laid out?

Assuming a contiguous storage area, my preferred approach would be something like your second bullet option, but to add a size value as the very first value in your configuration structure. Optionally (but not necessary since you now know the size of the struct), you can move the CRC/magic# to the second value in the struct, something like:

struct Config {
  uint32_t sz,    (or maybe size_t)
  uint32_t crc,
  ...
};

Now for any new version you know exactly how much data to read from the old configuration. The CRC being second is only a convenience but not necessary, since you could always do the math (sz-sizeof(crc)) to get to the CRC field at the end, but if you're going to rework things for the future, I'd consider moving it to second position and reading it directly as well.

Then any new configuration values for new versions are simply appended to the end and the size+CRC is updated to reflect that.

EDIT: typo
Siwastaja:
Magic number for the whole configuration which indicates the "revision" (magic number changes whenever large changes are made). Keep CRC a separate concept.

Increasing the configuration size at the end with something where all zeroes (or all ones) can be treated as "sane defaults" does not count as "large change", and hence does not require changing the magic number. This enables you to do maybe 95% of updates without breaking compatibility. Your usual feature creep.

When you find out you need to restructure old stuff, then you change the magic number and add a special migration code (which is quite PITA to write and test) which takes conf version xyz and turns it into conf version åäö. This has to understand what the conf means in order to generate the new, functionally equivalent conf. Maybe this is mostly copying variables into different locations.

To make your life easier, add dummy fields here and there unless you are very tight on flash. Have a convention so that unused bytes are written to 0 (or 0xff) and treat that as "sane default" in the new code. This way, you can avoid those migration updates.

Current project of mine is now on firmware version 9 and there has been one (1) update which required migration so far and that was back when we had some 3 customers or so, it was between releases 1 and 2. This scheme is not perfect by any means but it sure is simple.
nctnico:
I always add versions numbers to configuration data. I use a header that has size, version and a checksum (can be CRC or one of the SHA algorithms). By using a fixed header, I can use the same software to deal with any kind of configuration data and put it somewhere on a flash or eeprom memory. The higher layers can figure out how to deal with the various versions.
Navigation
Message Index
Next page
There was an error while thanking
Thanking...

Go to full version
Powered by SMFPacks Advanced Attachments Uploader Mod