Author Topic: Preserving settings over a firmware upgrade  (Read 3314 times)

0 Members and 1 Guest are viewing this topic.

Offline AndyC_772Topic starter

  • Super Contributor
  • ***
  • Posts: 4315
  • Country: gb
  • Professional design engineer
    • Cawte Engineering | Reliable Electronics
Preserving settings over a firmware upgrade
« on: July 25, 2023, 02:51:31 pm »
Hi,

I have a product which has been in production for a number of years, and which has a lot of configurable settings. Some of these are set at the factory and are hidden, while others are accessible to the end user.

The environment is an ARM microcontroller using internal storage only. No file system or networking, but it can be connected to a PC via a serial link.

Typically I store all of a product's settings in a C struct called 'nv' ('non-volatile'), which is stored in Flash whenever a change is made, and read back in as part of the boot process.

Within this struct there are all kinds of integers, strings, other structs and so on, and then right at the very end there's a uint32_t which must always be set to a 'magic number' to indicate that the rest of the struct has been populated with valid data. This number could be a fixed value, a CRC, a cryptographic hash, checksum, or whatever.

When the firmware boots, it checks for the magic number. If it's valid, then that's great, carry on as normal. If not, it assumes the configuration is broken, and restores factory default settings including a new magic number.

This works well. It ensures that the unit always boots with valid settings whether it's brand new and is booting for the first time, or if it's been updated to new firmware which (usually) has a larger 'nv' struct with the magic number in a different place.

Until quite recently, firmware updates have only ever been done at a very limited number of authorised sites, where having the unit reset to factory settings after an update isn't a problem. However, I'd now like to explore how to go about preserving settings when a unit is updated by the end user in the field.

The issue is, of course, that new firmware may well require some new settings that weren't supported before. Insert new items into a struct, and everything beyond them has to move.

A few options spring to mind, but I can't quite figure out which I dislike least.

  • Keep the original nv struct exactly the same forever. For new settings introduced in the next firmware release, create a new struct called nv2. Then nv3, nv4 etc, each with their own magic number to show they're valid.
  • As above, but add the new items to the end of nv so they sit after the magic number, which at least means I don't have to litter my code with nv3.this, nv5.that, nv2.the_other etc
  • Reserve space in nv between the last used setting and the magic number, put any new settings here. The magic number doesn't move, so it definitely has to be a CRC or hash, not a fixed value.
  • Tag the nv struct with a value that indicates its version number. Include start-up code that recognises previous versions and knows how to migrate settings to the current version.
  • Export settings in a readable format, with tags or commands to indicate what each value actually means. After an upgrade, reset to factory defaults, then parse the readable version to get back to a binary.
  • The better idea that I haven't thought of yet...!

Suggestions please, folks. What's the industry's solution to this problem?

Offline tszaboo

  • Super Contributor
  • ***
  • Posts: 8218
  • Country: nl
  • Current job: ATEX product design
Re: Preserving settings over a firmware upgrade
« Reply #1 on: July 25, 2023, 03:27:53 pm »
Think of it from the hardware perspective for a second. You can only erase sectors, that are for example 4KB in size, and you need to erase them to overwrite data in it. Is 4KB enough for you, than make the last 4 bytes eg. for your CRC and magic number, and place the data before, always.
How you organize your data before it... there are so many ways to do that I'm sure you find something you like. I think last time I was doing something like this, I had a lightweight version control for the config file, and the config version was the first data saved.
 

Offline Veteran68

  • Frequent Contributor
  • **
  • Posts: 727
  • Country: us
Re: Preserving settings over a firmware upgrade
« Reply #2 on: July 25, 2023, 03:56:40 pm »
Several ways to go about this, but just to clarify: are you considered concerned about the storage space available and ensuring it's contiguous, or are you simply concerned about the management of the code itself and how the struct is laid out?

Assuming a contiguous storage area, my preferred approach would be something like your second bullet option, but to add a size value as the very first value in your configuration structure. Optionally (but not necessary since you now know the size of the struct), you can move the CRC/magic# to the second value in the struct, something like:

struct Config {
  uint32_t sz,    (or maybe size_t)
  uint32_t crc,
  ...
};


Now for any new version you know exactly how much data to read from the old configuration. The CRC being second is only a convenience but not necessary, since you could always do the math (sz-sizeof(crc)) to get to the CRC field at the end, but if you're going to rework things for the future, I'd consider moving it to second position and reading it directly as well.

Then any new configuration values for new versions are simply appended to the end and the size+CRC is updated to reflect that.

EDIT: typo
« Last Edit: July 25, 2023, 05:37:57 pm by Veteran68 »
 

Offline Siwastaja

  • Super Contributor
  • ***
  • Posts: 9336
  • Country: fi
Re: Preserving settings over a firmware upgrade
« Reply #3 on: July 25, 2023, 04:06:48 pm »
Magic number for the whole configuration which indicates the "revision" (magic number changes whenever large changes are made). Keep CRC a separate concept.

Increasing the configuration size at the end with something where all zeroes (or all ones) can be treated as "sane defaults" does not count as "large change", and hence does not require changing the magic number. This enables you to do maybe 95% of updates without breaking compatibility. Your usual feature creep.

When you find out you need to restructure old stuff, then you change the magic number and add a special migration code (which is quite PITA to write and test) which takes conf version xyz and turns it into conf version åäö. This has to understand what the conf means in order to generate the new, functionally equivalent conf. Maybe this is mostly copying variables into different locations.

To make your life easier, add dummy fields here and there unless you are very tight on flash. Have a convention so that unused bytes are written to 0 (or 0xff) and treat that as "sane default" in the new code. This way, you can avoid those migration updates.

Current project of mine is now on firmware version 9 and there has been one (1) update which required migration so far and that was back when we had some 3 customers or so, it was between releases 1 and 2. This scheme is not perfect by any means but it sure is simple.
« Last Edit: July 25, 2023, 04:17:25 pm by Siwastaja »
 

Offline nctnico

  • Super Contributor
  • ***
  • Posts: 28429
  • Country: nl
    • NCT Developments
Re: Preserving settings over a firmware upgrade
« Reply #4 on: July 25, 2023, 04:17:10 pm »
I always add versions numbers to configuration data. I use a header that has size, version and a checksum (can be CRC or one of the SHA algorithms). By using a fixed header, I can use the same software to deal with any kind of configuration data and put it somewhere on a flash or eeprom memory. The higher layers can figure out how to deal with the various versions.
There are small lies, big lies and then there is what is on the screen of your oscilloscope.
 

Offline tom66

  • Super Contributor
  • ***
  • Posts: 7336
  • Country: gb
  • Electronics Hobbyist & FPGA/Embedded Systems EE
Re: Preserving settings over a firmware upgrade
« Reply #5 on: July 25, 2023, 05:13:06 pm »
Build the data in the memory a little like a key-value struct, where each setting has a 'key ID' (this can just be decoded to flash address) and a size.

The key ID can encode the type and size of variable.  For instance, the first byte can signal type (int8, uint16, float) and the second byte might encode maximum size.  Then the remaining 16 bits (assuming a 32-bit CPU) can effectively encode the address of the value.

Then you write a number of functions that do things like:

uint8_t getKeyAsU8(uint32_t key);
uint16_t getKeyAsU16(uint32_t key);


I don't know if you need to store strings.  That can get tricky as they are variable length, and they might need to be packed in a different way.

You could make it so that trying to get a key as an invalid type (e.g. can't convert U16 to U8) returns zero, throws exception (in C++) or passes an error state by reference. 

Future programs could access keys that have not yet been saved.  This can be handled by robust checks on the values, or by checking a version number and preloading values with defaults if they don't match.  If you go for the 'header' for each value, then the value can be loaded with a default if it is not yet set.

This method is similar to ublox's method for their flash configuration, though they do note in other app notes that firmware updates do not guarantee configuration will be kept in Flash.
 
The following users thanked this post: thm_w

Offline Siwastaja

  • Super Contributor
  • ***
  • Posts: 9336
  • Country: fi
Re: Preserving settings over a firmware upgrade
« Reply #6 on: July 25, 2023, 05:17:14 pm »
The id-length-data scheme has an advantage that you can update settings (not unlimited times of course) without erase cycle - just append {id-length-data} at the end. Flash erase state (usually 0xff) signifies end-of-chain. To fetch a value, scan the whole chain and keep the last value for each id (you can do this at boot, plus during update, and just maybe update a pointer to each item). It's also quite efficient in storage space when there are different size items. If you then need to add a new feature, no problem, you just add a new ID for it; it can be physically at any memory address.
 

Offline nctnico

  • Super Contributor
  • ***
  • Posts: 28429
  • Country: nl
    • NCT Developments
Re: Preserving settings over a firmware upgrade
« Reply #7 on: July 25, 2023, 05:38:19 pm »
Next step is to store the config as a JSON formatted text blob. But you still might need to do some kind of conversion if a single setting becomes two settings or vice versa.
There are small lies, big lies and then there is what is on the screen of your oscilloscope.
 

Offline tom66

  • Super Contributor
  • ***
  • Posts: 7336
  • Country: gb
  • Electronics Hobbyist & FPGA/Embedded Systems EE
Re: Preserving settings over a firmware upgrade
« Reply #8 on: July 25, 2023, 05:59:37 pm »
Next step is to store the config as a JSON formatted text blob. But you still might need to do some kind of conversion if a single setting becomes two settings or vice versa.

If you've got a bigger applications processor (we use Zynq's a lot) then JSON absolutely makes sense, it's quite efficient to parse, is human readable, error resistant (invalid keys can return defaults or throw exceptions on most parsers) and handles backwards and forwards compatibility reasonably well.
 
The following users thanked this post: tooki

Offline nctnico

  • Super Contributor
  • ***
  • Posts: 28429
  • Country: nl
    • NCT Developments
Re: Preserving settings over a firmware upgrade
« Reply #9 on: July 25, 2023, 07:28:45 pm »
You can use JSON on a small microcontroller just fine. In the end you only need a few simple text parsing functions to take the JSON data apart and extract the data you need.
There are small lies, big lies and then there is what is on the screen of your oscilloscope.
 

Offline AndyC_772Topic starter

  • Super Contributor
  • ***
  • Posts: 4315
  • Country: gb
  • Professional design engineer
    • Cawte Engineering | Reliable Electronics
Re: Preserving settings over a firmware upgrade
« Reply #10 on: July 25, 2023, 07:32:43 pm »
Some good ideas there folks, thank you all.

I think I'll go with:

- keep each new variant as similar as possible to what went before. Insert dummy variables where there used to be meaningful data, add anything new at the end. Everything stays in the same place as far as possible, at least until I run out of storage and have to do something drastic. Fortunately I have a lot more space available for code than I do for settings.

- add a revision number to the start of the nv struct, which increments each time there's a change that the code needs to know about. Come to think of it, I do this with register interfaces to FPGAs all the time, so it's hardly a foreign concept.

- move the magic number to the start, so it's always in the same place; we can still tell the difference between old config vs new config vs erased flash.

- at power up, check the revision number and run some one-time code to migrate settings as necessary. This might be as simple as setting factory defaults for new features, but it might include manipulating other data too. This is OK; better to update the settings once than forever have to cope with the fact that they might be stored in an out-of-date way.

Offline tom66

  • Super Contributor
  • ***
  • Posts: 7336
  • Country: gb
  • Electronics Hobbyist & FPGA/Embedded Systems EE
Re: Preserving settings over a firmware upgrade
« Reply #11 on: July 25, 2023, 07:33:45 pm »
You can use JSON on a small microcontroller just fine. In the end you only need a few simple text parsing functions to take the JSON data apart and extract the data you need.

The issue I forsee is updating one key possibly requires rewriting the whole JSON.  That possibly requires enough RAM to recreate the JSON as a string, then you need to overwrite all of the flash pages that change. If you only have a couple of settings, no issue, but a large number of settings quickly becomes a bottleneck.

Not an issue on a Zynq running Linux.  A headache on an STM32 with 32KB of RAM...
 

Offline nctnico

  • Super Contributor
  • ***
  • Posts: 28429
  • Country: nl
    • NCT Developments
Re: Preserving settings over a firmware upgrade
« Reply #12 on: July 25, 2023, 07:49:38 pm »
You can use JSON on a small microcontroller just fine. In the end you only need a few simple text parsing functions to take the JSON data apart and extract the data you need.
The issue I forsee is updating one key possibly requires rewriting the whole JSON.  That possibly requires enough RAM to recreate the JSON as a string, then you need to overwrite all of the flash pages that change. If you only have a couple of settings, no issue, but a large number of settings quickly becomes a bottleneck.

Not an issue on a Zynq running Linux.  A headache on an STM32 with 32KB of RAM...
No problem at all. You erase the flash page you want to write the settings to (which needs to be done anyway *) and then create (snprintf) each piece of the JSON data one by-one and commit to flash. In case the flash can be written in chunks of several bytes (which is typical for ECC flash), then all you need is a small extra buffer to capture data before you have a full flash chunk. You'd need maybe 100 bytes at most for buffering (on stack) to have long key names and reasonable long text strings for settings.

* Ofcourse you have to implement a double / multi buffering system in order to always retain a valid record with settings in case there is a power outage during write or the flash wears out.
There are small lies, big lies and then there is what is on the screen of your oscilloscope.
 

Offline Veteran68

  • Frequent Contributor
  • **
  • Posts: 727
  • Country: us
Re: Preserving settings over a firmware upgrade
« Reply #13 on: July 25, 2023, 08:03:36 pm »
My how things have changed since my early embedded systems days. Sure, compared to the bloat of something like XML, JSON is certainly much lighter, but it's still not what I'd call firmware light. I use it all the time in PC application code where you're storing or transferring lots of strings or human readable data, but I can't see incurring its data storage and parsing logic overhead for something like a firmware configuration.

High level markup languages are intended as self documenting data structures for data transfer and storage for high level business developers (many of which can barely code in Visual Basic or HTML/CSS/JS), not for an embedded systems developer dealing in bits and bytes, registers and interrupts. It's too easy to stream a series of bytes and lay them into a struct for use by the system. Then use the numerous tricks as discussed in this thread to manage the dynamic scalability of that data.

Then again the MCUs and SOCs we have today have mountains of storage and power compared to the old days when we counted every clock cycle and data bit, while hand-optimizing the assembler produced by C compilers (or just wrote it in ASM to begin with).

These kids today have it easy...  :P
 

Offline PlainName

  • Super Contributor
  • ***
  • Posts: 7509
  • Country: va
Re: Preserving settings over a firmware upgrade
« Reply #14 on: July 25, 2023, 09:18:51 pm »
If the user is applying the upgrade then that implies some kind of connection (USB, WiFi, BT), in which case cannot the device send the config to the PC or phone (or website) that's sending the upgrade? In a perfect world it could save the config on demand (even if that meant emailing it to the user) so it could be saved 'just in case', or restored after the user has screwed it up.

Although the on-chip config is no doubt fine as blob (actually a defined structure), converting to and from JSON for the purposes of saving and restoring would save lots of hassle when the format changes, perhaps by thing being inserted or removed.
 

Offline nctnico

  • Super Contributor
  • ***
  • Posts: 28429
  • Country: nl
    • NCT Developments
Re: Preserving settings over a firmware upgrade
« Reply #15 on: July 25, 2023, 09:22:33 pm »
If the user is applying the upgrade then that implies some kind of connection (USB, WiFi, BT), in which case cannot the device send the config to the PC or phone (or website) that's sending the upgrade? In a perfect world it could save the config on demand (even if that meant emailing it to the user) so it could be saved 'just in case', or restored after the user has screwed it up.

Although the on-chip config is no doubt fine as blob (actually a defined structure), converting to and from JSON for the purposes of saving and restoring would save lots of hassle when the format changes, perhaps by thing being inserted or removed.
Exactly! Conversion between configuration versions basically comes for free when using JSON to store the configuration.
There are small lies, big lies and then there is what is on the screen of your oscilloscope.
 

Offline Siwastaja

  • Super Contributor
  • ***
  • Posts: 9336
  • Country: fi
Re: Preserving settings over a firmware upgrade
« Reply #16 on: July 26, 2023, 05:57:31 am »
Exactly! Conversion between configuration versions basically comes for free when using JSON to store the configuration.

It only avoids the fixed memory offset relationships, but does not magically understand functional differences which still need migration code to solve. And if you have no functional differences, then surely you don't need to change the layout either. If you do like I suggested, add new stuff at the end, then the only case where layout changes are large, fundamental changes where functional changes are to be expected too. In which case you need migration code anyway.

Migration between two binary formats is actually simpler:
Code: [Select]
struct conf_v1
{
    int16_t thing1; // 1 LSB = 0.1
    int32_t thing2;
};

struct conf_v2
{
    double thing1; // add more range and resolution; fundamental change affecting both functionality and layout
    int32_t thing2;
}

void migrate_v1_to_v2(struct conf_v1 const * const v1, struct conf_v2 * const v2)
{
    v2->thing1 = v1->thing1 * 0.1;
    v2->thing2 = v1->thing2;
}

With json, this is otherwise the same but instead of direct access with variable names and assignment operators, you use the json library to parse and generate the json. No automation nor simplification here, only a complication.
« Last Edit: July 26, 2023, 06:00:11 am by Siwastaja »
 

Offline Siwastaja

  • Super Contributor
  • ***
  • Posts: 9336
  • Country: fi
Re: Preserving settings over a firmware upgrade
« Reply #17 on: July 26, 2023, 07:18:39 am »
I would use JSON to store the settings in flash when
1) the upstream, where the config is coming from, e.g. web UI or server, prefers JSON. Conversion into microcontroller memory layout has to be done somewhere; do it on the target itself and avoid multiple/intermediate formats
2) there is enough flash & RAM on the target to do so

Otherwise than that, it's the good old prefer data over code. If and when you can avoid conversions and directly use the memory layout and language syntax (in this case, probably C), do so. Some people love to add intermediate formats and write parsers for them, and it sure does help with job security because bugs sneak in and changes need modifications in multiple places - data structures themselves, parsers/generators, and you usually still can't avoid versioning.

What JSON actually provides? Two things basically:
A) interoperability with some web UI / database / server frameworks (mapping directly into their data structure types, that's the whole point)
B) decoupling of physical memory layout (lengths, offsets)

If you don't need JSON for A, then B can be achieved much more efficiently by id-length-data pattern described above. Instead of arbitrary length string key, you would use for example a 16-bit ID. But you don't usually need even that. It can be easily shown that the struct member assignment pattern shown in my above post is never more complex or more difficult than the JSON equivalent, where you always need to parse and generate every member and need to map (manually, or maybe with preprocessor stringification) from JSON key strings to variable names, and forgetting some is as disastrous as it is on the direct binary mapping. Worst thing is, you always have to do all of this parsing, even if you never change revisions, while on versioned structs you just do it with major changes. Assignment operator on C between the members of the two structs does all the memory layout work for you.

I can't really see any disadvantage on versioned structs. This is within the target itself so memory alignment and endianness are not even theoretical problems. Really it's the same struct where you would put the parsed JSON anyway / generate JSON from. Just write it as-is. RAM or FLASH, doesn't matter.
« Last Edit: July 26, 2023, 07:20:44 am by Siwastaja »
 

Offline PlainName

  • Super Contributor
  • ***
  • Posts: 7509
  • Country: va
Re: Preserving settings over a firmware upgrade
« Reply #18 on: July 26, 2023, 10:59:51 am »
Exactly! Conversion between configuration versions basically comes for free when using JSON to store the configuration.

It only avoids the fixed memory offset relationships, but does not magically understand functional differences which still need migration code to solve.
...
Migration between two binary formats is actually simpler:
Code: [Select]
void migrate_v1_to_v2(struct conf_v1 const * const v1, struct conf_v2 * const v2)
{
    v2->thing1 = v1->thing1 * 0.1;
    v2->thing2 = v1->thing2;
}

With json, this is otherwise the same but instead of direct access with variable names and assignment operators, you use the json library to parse and generate the json. No automation nor simplification here, only a complication.

Where you have that kind of migration code you're polluting the product for a one-time (possibly never) migration. And next time it won't work so you'll need another lot of migration code to do it all again. Or never again.

The idea with JSON (and it doesn't have to be JSON - that appears to be the format du jour, and previously might have been XML or INI or any other text-based thing) is that the parser on the product doesn't change to suit the data because the data is already OK. If there needs to be a migration you can do that off-product on a PC or phone or something, which has significant benefits. One is that a really tricky conversion is just invisible (to the product) code. Doesn't matter if it takes 2 weeks of an I7 to do it, or 20GB of RAM to store intermediate results. OK, a bit hyperbolic but that simple int to double may not be as simple as a 0.1 multiplier, particularly at the top and bottom of the range.

Another benefit is that you can do partial export and restore - just do a couple of values instead of the entire flash. Think registry, and how you can alter bits without having to write the multi-MB monstrosity it is, and that's because it's no a binary copy/modify thing.

Finally, and I'm sure we've all been there at some point, if all else fails you can manually create/modify the data to be changed/set. And it needs no support on the product beyond the standard save/restore function that doesn't change.

Edit: Forgot to say that in most cases a conversion wouldn't be necessary in the first place. JSON (or whatever) says 'this_var' is 76 and the product code just puts 76 into whatever it uses as store: int, double, long, string, who knows or cares. So 'coversion' is easy: write out int as "76", read "76" into double. And the user can alter that to 74 without know or caring what the storage slot is, or where it is.
« Last Edit: July 26, 2023, 12:04:57 pm by PlainName »
 

Offline Siwastaja

  • Super Contributor
  • ***
  • Posts: 9336
  • Country: fi
Re: Preserving settings over a firmware upgrade
« Reply #19 on: July 26, 2023, 01:09:33 pm »
Where you have that kind of migration code you're polluting the product for a one-time (possibly never) migration. And next time it won't work so you'll need another lot of migration code to do it all again. Or never again.

That is something you cannot completely avoid. Usage of JSON does not change this the slightest. You are just pushing the problem elsewhere: now you have to implement parsing, generation and mapping to internal variables anyway, even without any migration. This is at least the same or more work than the trivial case of migration using old and new structs where variables are assigned to each other.

For more complex cases of migration, where functionality changes enough to prevent simple 1:1 mapping, then both approaches (direct struct vs. json intermediate) need similar amount of "throw-away" code for migration. By adding mystical cargo cult layers, you can hide this work, but finally someone has to do it somewhere.

Remember we are discussing storage within a product, not interchange format. It's not like the CPU architecture (endianness) is changing. If the product also needs a configuration interchange format in ASCII for a server / web UI, then I wholeheartedly agree that JSON is one of the sanest choices, and if you just have resources on the MCU, bring it down to flash storage without another intermediate format, parse in one place.

Quote
If there needs to be a migration you can do that off-product on a PC or phone or something, which has significant benefits.

I agree, migration can be done on a server or something, it depends on the case which is the best, but throwaway code on an MCU is not necessarily that bad. If you have say 20 parameters needing remapping between two structs, that would be ~25 LoC with maybe 200 bytes of code generated. Much less than a JSON library.

Generally it's a good idea to use tools for the intended purpose, JSON for data interchange; it's excessive for memory location independent code if that is the only specific need.

Quote
Another benefit is that you can do partial export and restore - just do a couple of values instead of the entire flash. Think registry, and how you can alter bits without having to write the multi-MB monstrosity it is, and that's because it's no a binary copy/modify thing.
I don't think you understand how the flash works. It has to be erased by sector and fully rewritten. There are ways to circumvent that to some extent like the accumulating id-length-data pattern I described above, but that's even less suitable territory to force JSON to.
« Last Edit: July 26, 2023, 01:12:54 pm by Siwastaja »
 

Online DavidAlfa

  • Super Contributor
  • ***
  • Posts: 6403
  • Country: es
Re: Preserving settings over a firmware upgrade
« Reply #20 on: July 26, 2023, 02:45:26 pm »
You could keep older settings struct definitions, then detect, check and import them.
All the functions would need to be aware of older versions and how to handle them.
A simple example, might contains some errors and possibly a lot of alignment issues, this is only a simple idea:

Code: [Select]
#define SETTINGS_VERSION settings_v2            // Current settings version

typedef enum {
  settings_v0 = 0xFF00,
  settings_v1 = 0xFE01,
  settings_v2 = 0xFD02,
}settings_version;

typedef struct {          // First int is always settings version
  struct{
    uint16_t  version;
    uint16_t  data0;
    uint16_t  data1;
  }data;
  uint16_t  checksum;
} settings_v0_t;

typedef struct {
  struct{
    uint16_t  version;
    uint16_t  data0;
    uint16_t  data1;
    uint16_t  data2;
  }data;
  uint16_t  checksum;
} settings_v1_t;

typedef struct {
  struct{
    uint16_t  version;
    uint16_t  data0;
    uint16_t  data1;
    uint16_t  data2;
    uint16_t  data3;
  }data;
  uint16_t  checksum;
} settings_v2_t;

Code: [Select]
void init(void){
  void * settings = some_flash_address;
  uint16_t version = *(uint16_t*)settings;
  if(!check_settings(settings))
    if(version != SETTINGS_VERSION)
      import_settings(settings);
  else
    reset_settings();
}
Code: [Select]
// Return 1 if error, 0 if OK.
bool check_settings(void * settings){
  uint16_t new_checksum = checksum(settings);
  uint16_t checksum=0;

  switch (*(uint16_t*)settings){

    case settings_v0:
      checksum = (settings_v0_t*)settings->checksum;
      break;

    case settings_v1:
      checksum = (settings_v1_t*)settings->checksum;
      break;

    case settings_v2:
      checksum = (settings_v2_t*)settings->checksum;
      break;

    default:
      return 1;
  }
  if(new_checksum != checksum)
    return 1;
  return 0;
}
Code: [Select]
// Import different system settings, reset new values to their default state
void import_settings(void * settings){
  settings_v2_t  s2;                                         // Temporal buffer to import settings into
  switch (*(uint16_t*)settings){

    case settings_v0:
      s2.data.data0 = *(settings_v0_t*)settings->data.data0;
      s2.data.data1 = *(settings_v0_t*)settings->data.data1;
      s2.data.data2 = DEFAULT_DATA_2;
      s2.data.data3 = DEFAULT_DATA_3;     
      break;

    case settings_v1:
      s2.data.data0 = *(settings_v0_t*)settings->data.data0;
      s2.data.data1 = *(settings_v0_t*)settings->data.data1;
      s2.data.data2 = *(settings_v0_t*)settings->data.data2;
      s2.data.data3 = DEFAULT_DATA_3;
      break;

    case settings_v2:     // Nothing to do, same settigns version, we shouldn't reach this
      break;

    default:              // Unknown_settings_version !
      break;

  }

  s2.data.version = SETTINGS_VERSION;
  s2.checksum = checksum(s2);
  write_settings(s2);
  reboot();
}
Code: [Select]
// Compute checksum for each settings version
uint16_t checksum(void * settings){
  uint16_t sz=0;
  uint16_t checksum=0;
  uint8_t * data = (uint8_t*)settings;
 
  switch (*(uint16_t*)settings){   

    case settings_v0:
      sz = sizeof(settings_v0_t.data);
      break;

    case settings_v1:
      sz = sizeof(settings_v1_t.data);
      break;

    case settings_v2:
      sz = sizeof(settings_v2_t.data);
      break;

    default:
      return 0;
  }
  for(uint16_t i=0;i<sz;i++){
    checksum += *data++;
  }
  return checksum;
}
« Last Edit: July 28, 2023, 11:59:54 am by DavidAlfa »
Hantek DSO2x1x            Drive        FAQ          DON'T BUY HANTEK! (Aka HALF-MADE)
Stm32 Soldering FW      Forum      Github      Donate
 

Offline PlainName

  • Super Contributor
  • ***
  • Posts: 7509
  • Country: va
Re: Preserving settings over a firmware upgrade
« Reply #21 on: July 26, 2023, 03:46:51 pm »
Quote
I don't think you understand how the flash works. It has to be erased by sector and fully rewritten.

I am fully conversant with how to program flash, thank you, and ways to retain existing data whilst changing parts that are smaller than a flash page.
 

Offline thm_w

  • Super Contributor
  • ***
  • Posts: 7527
  • Country: ca
  • Non-expert
Re: Preserving settings over a firmware upgrade
« Reply #22 on: July 26, 2023, 09:41:47 pm »
Build the data in the memory a little like a key-value struct, where each setting has a 'key ID' (this can just be decoded to flash address) and a size.

The key ID can encode the type and size of variable.  For instance, the first byte can signal type (int8, uint16, float) and the second byte might encode maximum size.  Then the remaining 16 bits (assuming a 32-bit CPU) can effectively encode the address of the value.

Then you write a number of functions that do things like:
...

This is a cool example of what you are talking about for EEPROM but sadly not open source: https://www.realtime.bc.ca/articles/nonvol.html
Profile -> Modify profile -> Look and Layout ->  Don't show users' signatures
 

Offline nctnico

  • Super Contributor
  • ***
  • Posts: 28429
  • Country: nl
    • NCT Developments
Re: Preserving settings over a firmware upgrade
« Reply #23 on: July 28, 2023, 08:28:25 am »
Exactly! Conversion between configuration versions basically comes for free when using JSON to store the configuration.

It only avoids the fixed memory offset relationships, but does not magically understand functional differences which still need migration code to solve.
...
Migration between two binary formats is actually simpler:
Code: [Select]
void migrate_v1_to_v2(struct conf_v1 const * const v1, struct conf_v2 * const v2)
{
    v2->thing1 = v1->thing1 * 0.1;
    v2->thing2 = v1->thing2;
}

With json, this is otherwise the same but instead of direct access with variable names and assignment operators, you use the json library to parse and generate the json. No automation nor simplification here, only a complication.

Where you have that kind of migration code you're polluting the product for a one-time (possibly never) migration. And next time it won't work so you'll need another lot of migration code to do it all again. Or never again.
That is the cost of backward compatibility. For sure everyone will agree that it is better to avoid needing such migrations but either way it is better to prepare the way the data is stored so that software can detect the settings are compatible or not. When the configuration data is incompatible between versions, the next question is whether to use sensible defaults or try to convert.


There are small lies, big lies and then there is what is on the screen of your oscilloscope.
 
The following users thanked this post: Siwastaja

Offline Shonky

  • Frequent Contributor
  • **
  • Posts: 298
  • Country: au
Re: Preserving settings over a firmware upgrade
« Reply #24 on: July 28, 2023, 11:16:32 am »
Build the data in the memory a little like a key-value struct, where each setting has a 'key ID' (this can just be decoded to flash address) and a size.

The key ID can encode the type and size of variable.  For instance, the first byte can signal type (int8, uint16, float) and the second byte might encode maximum size.  Then the remaining 16 bits (assuming a 32-bit CPU) can effectively encode the address of the value.

Then you write a number of functions that do things like:
...

This is a cool example of what you are talking about for EEPROM but sadly not open source: https://www.realtime.bc.ca/articles/nonvol.html
Full source linked at the end?
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf