How to add in-field firmware upgrade

[Saimoun]

Ha ha ha I was literally about to ask, how well does the instruction "DO NOT POWER OFF" work... 

But yea then that might just be easier, to keep the bootloader at the beginning of the flash. My MCU has pages of 1kB so it's all good.
Also I had this idea, *if* ever one day I have no choice but to update the bootloader, I could:

• Make a special firmware which the user will load via the normal bootloader
• This special firmware, when run, will update the bootloader (after a press of a button f.x.) - that is critical but it will be very quick, just moving bytes from one part of the flash to the bootloader area
  
• Then the user can load a new firmware or the original one using the updated bootloader.

Anyways thank you all - super helpful asking here as always  

[ejeffrey]

Yes that is the way to do it.  You can reduce but not eliminate the window of vulnerability if you are careful. But the question is: if your bootloader is able to successfully load the special firmware why are you updating it?

[Peabody]

A while back I wrote an SD card bootloader for MSP430 processors.  The bootloader was attached to a normal program flashed using JTAG with a Launchpad.  Then when run, that program installed the bootloader at the beginning of flash memory, and set the startup vector to point to the bootloader.  After that, on powerup if no SD card was present, it would jump to the first byte after the bootloader.  But if a card was present, it would search the root directory for the latest file of the right size, then read in its contents and flash it, then shut down.  Everything was done in assembler, and the bootloader used exactly 1K, which was a convenient size.  I found that locating various pieces at the right places was easier in assembler without having to fight with linkers and such.

Then I had to write a Windows program that would read in an IntelHex file prepared by an assembler or compiler, run various checks to make sure everything was right, then output the appropriate binary file that would be copied to the SD card.  I was planning to distribute the firmware updates by sending users a new binary file, which they would copy to their SD cards, then install the update.

It all worked, but I never got to use it in a project.  I used the SD card because I wanted to be able to update in the field - literally - with no computer or mains power at the remote site.  Just carry the SD card to the site, and you're in like Flynn.

I think there are a number of ways this can be done.  I have mixed feelings about assembler vs compiler.  You don't want to have to learn assembler for a particular MCU, but on the other hand, I've never had much luck getting a compiler or linker to put things where I want them.  It's probably just all in knowing how.

[Siwastaja]

Yes that is the way to do it.  You can reduce but not eliminate the window of vulnerability if you are careful. But the question is: if your bootloader is able to successfully load the special firmware why are you updating it?

Indeed, plan the bootloader well and test is thoroughly and chances are very high you don't need to ever update it.

Alternating between two application images makes the device non-brickable - it will always boot with valid application even if power is lost during flash erase-write cycle. This requires at least 3 flash sectors, and enough flash memory for (2*app) + bootloader. You can make the bootloader check which app has a valid checksum plus a later version number, and boot that one, for example.

Often an acceptable solution is to have bootloader and one copy of application. With power loss during erase-write cycle, the application becomes non-bootable, but bootloader stays intact so trying flashing again will be possible. 2 flash sectors and enough memory for 1*app + bootloader is needed.

1 sector + app only with flasher code integrated requires least memory, is easiest to develop and use, but carries risk of brickage. If you can't fit the whole fw image in RAM at once but need to communicate + write piece by piece, then this risk becomes way too high IMO. But if you can buffer the whole image, the risk might be manageable.

[tellurium]

To add to what has been said already.

In my company, we used to deal with OTA firmware updates a lot. We used a bootloader and 2 flash partitions for an app.
In our case, customers also stored some configuration data on flash, which was part of an app partition.

So, sometimes a new image was perfectly fine, all checksums are OK. But their custom configuration, for one or another reason, bricked a device. The same binary worked fine on 1000 devices, but bricked the 1001st.

The solution for that was a to use a "blessed" marker, somewhere in the flash. A current firmware image is "blessed", a new one is not. The bootloader flashes a new app partition, verifies checksums, and reboots - into that NEW partition, which is not yet marked as blessed. At the same time, it starts a hardware timer that simply reboots the device in X amount of seconds. That X seconds should be enough to boot, initialize, verify the new functionality, and "bless" a new partition manually by some user action - so on the next reboot, a new firmware gets booted.

If a device hangs, crashes, or whatever - a HW timer restarts a device, and it reboots back to the old "blessed" partition.

This way, we used to OTA even very dodgy, bad firmwares in field, and that mechanism was always rebooting the good firmware back on any trouble - for example, if we lost connection with a device.

[dkonigs]

It all worked, but I never got to use it in a project.  I used the SD card because I wanted to be able to update in the field - literally - with no computer or mains power at the remote site.  Just carry the SD card to the site, and you're in like Flynn.

I think there are a number of ways this can be done.  I have mixed feelings about assembler vs compiler.  You don't want to have to learn assembler for a particular MCU, but on the other hand, I've never had much luck getting a compiler or linker to put things where I want them.  It's probably just all in knowing how.

It really comes down to learning how to muck with the linker configuration file.  For simply defining where the code as a whole goes, this is often enough.

If you want to place specific functions, data structures, or data blobs, then there's section attributes on C code and tools like GNU objcopy (if you need to reorganize the output).  And of course, looking at your build process output with a hex editor to verify that it behaved the way you want.  This is a bit frustrating, but with enough trial-and-error can be made to work.

For one of my projects, I have a bootloader that installs firmware updates off a USB stick (device has a USB host port).  Functionally similar to what you're probably doing with an SD card.  For another project, where the device normally acts as a USB peripheral, I'm using UF2.  But in either case, the higher level design is similar.  Bootloader sits in a small area at the start of flash, and depending on how its started it'll either update the real firmware or simply jump to it.

[Saimoun]

Thank you all for the feedback - always good to hear what others have done

Yes that is the way to do it.  You can reduce but not eliminate the window of vulnerability if you are careful. But the question is: if your bootloader is able to successfully load the special firmware why are you updating it?

Well the whole point in this case *is* to update the bootloader. But yes, like Siwastaja said the idea is to plan it and test it well, and never have to update it (but it's nice to know it's still a possiblity! ).

1 sector + app only with flasher code integrated requires least memory, is easiest to develop and use, but carries risk of brickage. If you can't fit the whole fw image in RAM at once but need to communicate + write piece by piece, then this risk becomes way too high IMO. But if you can buffer the whole image, the risk might be manageable.

1 sector for bootloader + 1 app only seems fine to me, if the upgrade fails the user can just try again since the bootloader will be intact, correct?
But I am not sure what brickage is so maybe I'm missing something here
Do you mean if the app image is badly loaded? Then the checksum will fail and the user can simply do it again - or is the worry here that the checksum keeps failing so that the user can actually never get to use his device again?

[Siwastaja]

Brickage is power failure during erase-write operation. The problem with typical flash is, erase is SLOW! Writing is of course slow too.

For example, erasing and writing a sector on STM32H7 device seems to take around a second with maximum possible erase/write parallelism setting and all data available in RAM.

Once you have started the erase, data is lost. Subsequent write must succeed. You can remove the risk of software bug by making the erase-write part extremely simple (a few lines of code running with interrupts disabled), and you will of course do CRC check before starting that erase. But this still leaves the power loss risk.

Yes, 1 bootloader + 1 app (on different sectors* of course) can be made in such way that even if app update fails, bootloader stays intact so user will have a system which does not boot to the app, but still has some way of retrying the update, but maybe not through a menu you would usually use, if that part is within app.

1 bootloader + 2 app has the advantage that the actual fully performing application can always boot even if update fails. This may or may not be important. I don't think it is, you can just give instructions with the firmware update, "in rare case of failure during update, do this special sequence to try again". Or maybe there is no special sequence at all, maybe the user always updates the firmware through a mechanism provided by the bootloader alone, simplest would be "insert media with the fw update file and reboot" or "start this PC software which floods update packets through interface X and reboot", bootloader then checking the availablity and correctness of image and performing the update.
 

*) sector here meaning "smallest eraseable unit", sometimes called page

[PCB.Wiz]

But hey, I missed you use a GD32 MCU. I haven't used them, but big kids on the streets talked about these things shadowing the whole flash into RAM at boot and run out of RAM. If this is really the case, then you would not need to worry at all, just write code that does whatever with the flash.

This sounds crazy - never heard about this... You mean the MCU itself would do that? I've spent a lot of time going through many GD32 Manuals, never saw something like this.
How would that even work since the RAM is much smaller (especially if your variables already use 90% of it)?

Even if some parts did do this, you need caution.

IC vendors have used this approach on experimental/alpha parts,  as the FAB sample costs for not-flash process are lower.
So they expect multiple die iterations and choose the lowest iteration cost, then as the part is proven, they might change to a more normal true flash design.

So you cannot guarantee a non documented feature, remains in the part.

[peter-h]

I wrote a boot loader for the 32F417 and Simon you are welcome to the source. In fact I probably posted it here a year or so ago; I ran some very long threads on this topic.

It isn't all that easy to do it. Quite a learning curve and most of the time when it doesn't work you have to reload everything with the debugger

Apparently one can flash "from the flash" because the CPU supposedly doesn't crash (it supposedly gets a million wait states but it all still runs) but I didn't do that. I set up a 32k buffer in RAM and that enabled me to flash the "boot block" (bottom 32k in my product) in one go; it takes about 300ms and that is the entire "bricking time window".

Important that the new firmware is stored locally and CRC validated, with a post build step in your devkit to add the CRC at the end of the firmware, so this can be checked. If you don't have enough local storage for the entire CPU FLASH then you are taking a risk.

Other approaches involve never flashing the bottom 32k, etc and keeping all the relevant code, including the RAM-executed boot loader, in there. I spent a lot of time on this...

Obviously you need to limit the risk of all your customers loading some broken firmware at the same time. That will likely sink your company. Phone companies minimise this by scheduling regional updates.