General > General Technical Chat

Preserving settings over a firmware upgrade

<< < (4/8) > >>

nctnico:

--- Quote from: PlainName on July 25, 2023, 09:18:51 pm ---If the user is applying the upgrade then that implies some kind of connection (USB, WiFi, BT), in which case cannot the device send the config to the PC or phone (or website) that's sending the upgrade? In a perfect world it could save the config on demand (even if that meant emailing it to the user) so it could be saved 'just in case', or restored after the user has screwed it up.

Although the on-chip config is no doubt fine as blob (actually a defined structure), converting to and from JSON for the purposes of saving and restoring would save lots of hassle when the format changes, perhaps by thing being inserted or removed.

--- End quote ---
Exactly! Conversion between configuration versions basically comes for free when using JSON to store the configuration.

Siwastaja:

--- Quote from: nctnico on July 25, 2023, 09:22:33 pm ---Exactly! Conversion between configuration versions basically comes for free when using JSON to store the configuration.

--- End quote ---

It only avoids the fixed memory offset relationships, but does not magically understand functional differences which still need migration code to solve. And if you have no functional differences, then surely you don't need to change the layout either. If you do like I suggested, add new stuff at the end, then the only case where layout changes are large, fundamental changes where functional changes are to be expected too. In which case you need migration code anyway.

Migration between two binary formats is actually simpler:

--- Code: ---struct conf_v1
{
    int16_t thing1; // 1 LSB = 0.1
    int32_t thing2;
};

struct conf_v2
{
    double thing1; // add more range and resolution; fundamental change affecting both functionality and layout
    int32_t thing2;
}

void migrate_v1_to_v2(struct conf_v1 const * const v1, struct conf_v2 * const v2)
{
    v2->thing1 = v1->thing1 * 0.1;
    v2->thing2 = v1->thing2;
}

--- End code ---

With json, this is otherwise the same but instead of direct access with variable names and assignment operators, you use the json library to parse and generate the json. No automation nor simplification here, only a complication.

Siwastaja:
I would use JSON to store the settings in flash when
1) the upstream, where the config is coming from, e.g. web UI or server, prefers JSON. Conversion into microcontroller memory layout has to be done somewhere; do it on the target itself and avoid multiple/intermediate formats
2) there is enough flash & RAM on the target to do so

Otherwise than that, it's the good old prefer data over code. If and when you can avoid conversions and directly use the memory layout and language syntax (in this case, probably C), do so. Some people love to add intermediate formats and write parsers for them, and it sure does help with job security because bugs sneak in and changes need modifications in multiple places - data structures themselves, parsers/generators, and you usually still can't avoid versioning.

What JSON actually provides? Two things basically:
A) interoperability with some web UI / database / server frameworks (mapping directly into their data structure types, that's the whole point)
B) decoupling of physical memory layout (lengths, offsets)

If you don't need JSON for A, then B can be achieved much more efficiently by id-length-data pattern described above. Instead of arbitrary length string key, you would use for example a 16-bit ID. But you don't usually need even that. It can be easily shown that the struct member assignment pattern shown in my above post is never more complex or more difficult than the JSON equivalent, where you always need to parse and generate every member and need to map (manually, or maybe with preprocessor stringification) from JSON key strings to variable names, and forgetting some is as disastrous as it is on the direct binary mapping. Worst thing is, you always have to do all of this parsing, even if you never change revisions, while on versioned structs you just do it with major changes. Assignment operator on C between the members of the two structs does all the memory layout work for you.

I can't really see any disadvantage on versioned structs. This is within the target itself so memory alignment and endianness are not even theoretical problems. Really it's the same struct where you would put the parsed JSON anyway / generate JSON from. Just write it as-is. RAM or FLASH, doesn't matter.

PlainName:

--- Quote from: Siwastaja on July 26, 2023, 05:57:31 am ---
--- Quote from: nctnico on July 25, 2023, 09:22:33 pm ---Exactly! Conversion between configuration versions basically comes for free when using JSON to store the configuration.

--- End quote ---

It only avoids the fixed memory offset relationships, but does not magically understand functional differences which still need migration code to solve.
...
Migration between two binary formats is actually simpler:

--- Code: ---void migrate_v1_to_v2(struct conf_v1 const * const v1, struct conf_v2 * const v2)
{
    v2->thing1 = v1->thing1 * 0.1;
    v2->thing2 = v1->thing2;
}

--- End code ---

With json, this is otherwise the same but instead of direct access with variable names and assignment operators, you use the json library to parse and generate the json. No automation nor simplification here, only a complication.

--- End quote ---

Where you have that kind of migration code you're polluting the product for a one-time (possibly never) migration. And next time it won't work so you'll need another lot of migration code to do it all again. Or never again.

The idea with JSON (and it doesn't have to be JSON - that appears to be the format du jour, and previously might have been XML or INI or any other text-based thing) is that the parser on the product doesn't change to suit the data because the data is already OK. If there needs to be a migration you can do that off-product on a PC or phone or something, which has significant benefits. One is that a really tricky conversion is just invisible (to the product) code. Doesn't matter if it takes 2 weeks of an I7 to do it, or 20GB of RAM to store intermediate results. OK, a bit hyperbolic but that simple int to double may not be as simple as a 0.1 multiplier, particularly at the top and bottom of the range.

Another benefit is that you can do partial export and restore - just do a couple of values instead of the entire flash. Think registry, and how you can alter bits without having to write the multi-MB monstrosity it is, and that's because it's no a binary copy/modify thing.

Finally, and I'm sure we've all been there at some point, if all else fails you can manually create/modify the data to be changed/set. And it needs no support on the product beyond the standard save/restore function that doesn't change.

Edit: Forgot to say that in most cases a conversion wouldn't be necessary in the first place. JSON (or whatever) says 'this_var' is 76 and the product code just puts 76 into whatever it uses as store: int, double, long, string, who knows or cares. So 'coversion' is easy: write out int as "76", read "76" into double. And the user can alter that to 74 without know or caring what the storage slot is, or where it is.

Siwastaja:

--- Quote from: PlainName on July 26, 2023, 10:59:51 am ---Where you have that kind of migration code you're polluting the product for a one-time (possibly never) migration. And next time it won't work so you'll need another lot of migration code to do it all again. Or never again.
--- End quote ---

That is something you cannot completely avoid. Usage of JSON does not change this the slightest. You are just pushing the problem elsewhere: now you have to implement parsing, generation and mapping to internal variables anyway, even without any migration. This is at least the same or more work than the trivial case of migration using old and new structs where variables are assigned to each other.

For more complex cases of migration, where functionality changes enough to prevent simple 1:1 mapping, then both approaches (direct struct vs. json intermediate) need similar amount of "throw-away" code for migration. By adding mystical cargo cult layers, you can hide this work, but finally someone has to do it somewhere.

Remember we are discussing storage within a product, not interchange format. It's not like the CPU architecture (endianness) is changing. If the product also needs a configuration interchange format in ASCII for a server / web UI, then I wholeheartedly agree that JSON is one of the sanest choices, and if you just have resources on the MCU, bring it down to flash storage without another intermediate format, parse in one place.


--- Quote ---If there needs to be a migration you can do that off-product on a PC or phone or something, which has significant benefits.
--- End quote ---

I agree, migration can be done on a server or something, it depends on the case which is the best, but throwaway code on an MCU is not necessarily that bad. If you have say 20 parameters needing remapping between two structs, that would be ~25 LoC with maybe 200 bytes of code generated. Much less than a JSON library.

Generally it's a good idea to use tools for the intended purpose, JSON for data interchange; it's excessive for memory location independent code if that is the only specific need.


--- Quote ---Another benefit is that you can do partial export and restore - just do a couple of values instead of the entire flash. Think registry, and how you can alter bits without having to write the multi-MB monstrosity it is, and that's because it's no a binary copy/modify thing.

--- End quote ---
I don't think you understand how the flash works. It has to be erased by sector and fully rewritten. There are ways to circumvent that to some extent like the accumulating id-length-data pattern I described above, but that's even less suitable territory to force JSON to.

Navigation

[0] Message Index

[#] Next page

[*] Previous page

There was an error while thanking
Thanking...
Go to full version
Powered by SMFPacks Advanced Attachments Uploader Mod