STM32 code that flashes new code?

[n4te]

In my ESP32 projects I have a mechanism where I send firmware bytes, write them to flash, reboot and my new code is running. Now I want to do that same thing on STM32.

In STM32 I have to deal with the flash directly. That is fine, but code is executed from flash and I can't overwrite the code that is running. I thought about putting my update functions in RAM, but that doesn't work since I need to call functions like HAL_FLASH_Program that wouldn't be in RAM and call many other functions.

The ESP32 apparently writes to a different location, then changes the entry point and reboots. I'm using STM32F411CEU6, which doesn't appear to have this "dual bank" functionality. Is that right?

What do people typically do? I know there's a bootloader, but it seems to want to handle everything itself, eg it reads from UART. I'd rather my code do the receiving, though I could pass the bytes off to something else. I could give that up, but I think I'd need to manipulate the boot pins to use the bootloader. Is that right? I did not plan that in my hardware and I'd not like to do yet another hardware revision.

Should I write my own bootloader? Can I do that without a hardware change? Is there another way?

EDIT: Solved!

[SiliconWizard]

Executing from RAM would be one option if you want to be able to write the entire flash memory without any reserved area. Don't use the HAL for that. If you are really not comfortable writing the flash functions yourself, you could always copy the source code (it's 100% available and free to use) of the HAL flash functions, wrap it in your own functions and put these in a RAM section with the appropriate attribute.

Otherwise, yes the most common approach is to write a "bootloader". It doesn't really have to be a bootloader in the true sense - a bootloader is usually what gets executed upon reset, and either jumps into your main user code, or handles firmware update, depending on some condition (like a GPIO state or something like that.)

But you can just reserve a small flash area for storing your firmware update functions and call them from your firmware when you want to trigger a firmware update. This will require modifying the linker script to create a section reserved for this piece of code, and assign said functions to the corresponding section, that will be separate from the rest of the flash where your actual firmware code lies.

[ataradov]

It does not need to be a separate bootloader. Just a single function that is a part of the main application will do the trick.

No matter what you do, you will have to store the received firmware first. So, your firmware must use less than half of the flash or you will need external storage.

When you  receive and verify the firmware, you can call a small RAM function that would transfer the image into the main flash and reboot the device. Obviously all the dependencies like  HAL_FLASH_Program must be located in the RAM too, but there should not too many of them. And it may make sense to re-implement them in a more  compact way anyway.

A lot of this depends on your requirements for robustness and recovery from failed updates. It can get very complicated.

[n4te]

That all makes sense, thank you both!

Writing a "bootloader" that does the receiving and flashing is interesting, if a bit daunting to get the dependencies together to do that. I guess I'd create a separate CubeIDE project for the bootloader.

My code is ~30% of flash in release mode and won't get much larger. Using twice the flash and a RAM function to copy it over seems pretty easy. I'll try to make this approach work first, starting by trying to simplify the flash copy.

Cheers!

[boB]

You use a bootloader that stays where it is.

Then you receive bytes that says you are going to re-flash your application and the bootloader erases those sectors of flash and brings in the new code, say, 256 bytes at a time and transfers that to flash in order.

If needed, the bootloader can also decrypt the new code to flash so many bytes at a time

boB

[n4te]

@boB Say my data is coming in via UART. How do I hand over the UART reading to the bootloader?

[SiliconWizard]

One benefit of using the conventional "bootloader" approach (which is a piece of code that gets executed first thing upon reset) is that, unless the bootloader itself is buggy (but you can make it as simple and robust as possible), it makes the device virtually unbrickable, as there will always be a way for the user of reflashing the firmware if something goes wrong.

If the flashing code is part of your firmware, it's more prone to making the device brickable if a firmware update fails.

So, just one further thing to keep in mind no matter what approach you pick.

[ataradov]

daunting to get the dependencies together to do that

A function to copy a range from one part of the flash to another part of the flash is going to be less than 20 lines of pure C code with no other function calls. Just implement it directly without relying on HAL.

Managing separate project for a simple thing like this would be a different type of pain. The first one needs to be done once, this one will last forever.

Plus with the first approach, you will be distributing the update code with each new firmware, making fixes in that code easier if necessary.

But yes, with all in approach, if something goes wrong or the power cuts out during the update, your device is bricked. If robustness is a requirement, then bootloader should be separate.

[n4te]

My 10 devices go in a ceiling. This remote update allows me to not have to dig them out. Worst case if the update bricks it, I'd drag out a ladder and reprogram them manually. Given that I probably don't need as many safeguards as other applications.

I have cobbled together a beast! Sectors 0-5 are 256KB for the main program, sectors 6 and 7 are 256KB temp space where I write the update, then I call this:

Code: [Select]

#define SECTOR0 0x08000000
#define SECTOR6 0x08040000

#ifdef DEBUG
static __RAM_FUNC void ramLogChar (char c) { // ITM_SendChar
if (((ITM->TCR & ITM_TCR_ITMENA_Msk) != 0UL) && // ITM enabled
((ITM->TER & 1UL               ) != 0UL)) {  // ITM Port #0 enabled
#undef u32
while (ITM->PORT[0U].u32 == 0UL)
__NOP();
#define u32 uint32_t
#undef u8
ITM->PORT[0U].u8 = (uint8_t)c;
#define u8 uint8_t
}
}

static __RAM_FUNC void ramLogNum (u8 value) {
if (value >= 100) {
ramLogChar('1');
ramLogChar('0');
ramLogChar('0');
} else {
if (value >= 10) ramLogChar('0' + value / 10);
ramLogChar('0' + value % 10);
}
}
#else
#define ramLogChar(...)
#define ramLogNum(...)
#endif

static __RAM_FUNC void ramWaitForCompletion () { // FLASH_WaitForLastOperation
while (true)
if (__HAL_FLASH_GET_FLAG(FLASH_FLAG_BSY) == RESET) break;
if (__HAL_FLASH_GET_FLAG(FLASH_FLAG_EOP) != RESET) __HAL_FLASH_CLEAR_FLAG(FLASH_FLAG_EOP);
#if defined(FLASH_SR_RDERR)
if (__HAL_FLASH_GET_FLAG((FLASH_FLAG_OPERR | FLASH_FLAG_WRPERR | FLASH_FLAG_PGAERR |
FLASH_FLAG_PGPERR | FLASH_FLAG_PGSERR | FLASH_FLAG_RDERR)) != RESET) NVIC_SystemReset();
#else
if (__HAL_FLASH_GET_FLAG((FLASH_FLAG_OPERR | FLASH_FLAG_WRPERR | FLASH_FLAG_PGAERR | \
FLASH_FLAG_PGPERR | FLASH_FLAG_PGSERR)) != RESET) NVIC_SystemReset();
#endif
}

// Everything used by this function must be in RAM.
static __RAM_FUNC void ramCopy (u32 length) {
__disable_irq();

// Erase sectors 0 through 5 (FLASH_Erase_Sector).
for (u32 i = 0; i <= 5; i++) {
__HAL_IWDG_RELOAD_COUNTER(&hiwdg);

ramLogNum(i);
ramLogChar('\n');
CLEAR_BIT(FLASH->CR, FLASH_CR_PSIZE);
FLASH->CR |= FLASH_PSIZE_WORD;
CLEAR_BIT(FLASH->CR, FLASH_CR_SNB);
FLASH->CR |= FLASH_CR_SER | (i << FLASH_CR_SNB_Pos);
FLASH->CR |= FLASH_CR_STRT;
ramWaitForCompletion();
CLEAR_BIT(FLASH->CR, FLASH_CR_SER | FLASH_CR_SNB);
}

u8 nextPercent = 0;
for (u32 i = 0; i < length; i += 4) {
__HAL_IWDG_RELOAD_COUNTER(&hiwdg);

// Write to flash (HAL_FLASH_Program).
CLEAR_BIT(FLASH->CR, FLASH_CR_PSIZE);
FLASH->CR |= FLASH_PSIZE_WORD;
FLASH->CR |= FLASH_CR_PG;
*((__IO u32*)(SECTOR0 + i)) = *((u32*)(SECTOR6 + i));
ramWaitForCompletion();
FLASH->CR &= ~FLASH_CR_PG;

u8 percent = i * 100 / length;
if (percent >= nextPercent) {
nextPercent = percent + 1;
ramLogNum(percent);
ramLogChar('%');
ramLogChar('\n');
}
}
NVIC_SystemReset();
}

One goofy thing is that if anything goes wrong, all I can do is reboot. In that case it's probably bricked until I update it manually.

And now to test it!

Edit: Code updated and tested! It assumes the BIN file is divisible by 4 and that .RamFunc goes in RAM. It even has simple logging.

[ataradov]

Careful with HAL_GetTick(). It and all its dependencies must be located in the RAM. And I don't really see the point. If it times out, something is really broken. Like the device is dead.

Also, make sure to disable interrupts on entry.

[n4te]

Good idea, I added __disable_irq() and kicking the IWDG.

Oh I missed HAL_GetTick, good catch. I think I still need to wait for FLASH_FLAG_BSY. I guess I'll just busy wait and let the 30 second IWDG reboot.

I've updated the code in my last post.

Edit: I should probably get rid of memcpy. Done!

[radiolistener]

just put your config structure to separate section to avoid intersect with other data or code, so you can safely rewrite it. If you're using multi-threading, synchronize access to that structure to avoid data corruption or incorrect reading while other thread updates the data.

[boB]

@boB Say my data is coming in via UART. How do I hand over the UART reading to the bootloader?

Just put a UART driver in the bootloader code.

I have a product that uses either USB or UART for updating code.  Whichever one is active upon bootup.

BTW, my bootloader just waits 2 or 3 seconds when booted to see if there is an update.  You could also have the main app go to bootloader when a special  command is received or a button is pushed.

boB

[Jeroen3]

Usually you write to bootloader to the bootable area of the FLASH. Such that it boots, and then loads. Hence the name.
The bootloader verifies the integrity of the main program (CRC) and handover execution to the main program. (write the vector table remap register and set the PC to the main program).
When bad it will enter firmware update mode.

This is also how the ROM bootloader works.

This way, you should never end with a bricked device. However, there are some parts that make this more challanging:
- the main program must be linked for FLASH+offset because where it by default is linked to is now where the bootloader lives.
- the main program never starts if the bootloader isn't programmed.
- the bootloader can be a secondary project in your workspace that compiles using reduced linker file.
- you may need to do some stuff around STM32CubeIDE or compiler/linker such your main project also includes the bootloader data inside the final hex file.
- updating the bootloader must be done from within the main program. Typically this can be a small enough project that you can include an updated bootloader image as data within the main program.

Alternative if you can't move the main program location. An updater project can be compiled to live and run from RAM and be stored as data inside the main program.
The main program can then copy it to RAM and hand over execution. The updater can do all it wants. Drawback, on powerloss your device needs new SWD loading.
 

[Siwastaja]

One benefit of using the conventional "bootloader" approach (which is a piece of code that gets executed first thing upon reset) is that, unless the bootloader itself is buggy (but you can make it as simple and robust as possible), it makes the device virtually unbrickable, as there will always be a way for the user of reflashing the firmware if something goes wrong.

If the flashing code is part of your firmware, it's more prone to making the device brickable if a firmware update fails.

True only if the "usual" update route is through some very simple physical interface, like serial port, and your customers are fine doing the normal updates through this route. But this is rarely the case today. If you want over-the-air update, maybe even secure, then it has all the complexity of the application, and the related risks, and it's actually safer to have that as part of the app. If you then fuck up the app so it can't connect and update, then the device is "bricked" to the state of needing a physical cable - that would just be SWD/JTAG/etc. instead of say UART.

Today normal people won't have either simple serial port cables or SWD cables hanging around, so really you need to service the bricked devices, or supply your customer with an "update cable", no way around it. So I prefer the way of having a bootloader which does nothing else but checks CRC and boots, and then use the application to update the application. Otherwise, over-the-air updates would be impossible, or I would need updateable bootloaders which is then again a total mess.

I manage the risk by careful testing that whenever I release the new firmware, make damn sure it boots and accepts new firmware through the usual route; test that well, even if you must half-ass testing everything else.

(In some cases, I have had to work with devices which have either only one flash sector (eraseable unit) or too little flash to hold two applications. In such cases, update process always has power loss brickage risk. But it's the simplest case to design for; just make sure everything related to the update is fully in RAM, erase and write; job done. But when doing this stuff properly, you would always erase and write a different part in flash, and then you don't have to put the flasher code in RAM; just execute from flash as usual, because you are not going to erase that half.)

[tellurium]

If we're talking about the update over the UART, then please note that all STM32 have a built-in bootloader capable of updating.

Just reset the MCU in the bootloader mode (typically, pull boot0 high then reset), and use a standard STM32 bootloader protocol, described in AN3155.

There open source implementations out there that implement that, e.g. https://pypi.org/project/stm32loader/

[Siwastaja]

If we're talking about the update over the UART, then please note that all STM32 have a built-in bootloader capable of updating.

Yes. If an "update dongle" is acceptable, then you can just use the factory UART bootloader. I think it's quite a special edge case if you need something of your own, yet that something is so simple that you can trust it never has to be updated.

[n4te]

The "write to second half of flash, erase and copy from a RAM function" works! I've updated the code in my earlier post. It even has simple SWO logging which helped me find some issues (I couldn't get JLink to debug the RAM function). Just don't try to log a string unless you ensure it's in RAM.

The bootloader discussion is super interesting and I'll definitely keep it in mind for other projects. For this project the risk of a failed update is acceptable and I have the space to store 2 copies of the firmware, so I'll roll with the RAM function copy approach.

I CRC32 the data as it's sent in 2048 chunks, resending if a chunk fails. After writing to the second half of flash, I CRC32 the written data and abort if it fails. Otherwise the RAM function does the copy. As long as the RAM function doesn't fail, everything should be good.

To use the built-in bootloader I'd need a hardware change to control BOOT0. Also I feel like if something goes wrong doing that, I'd be at a loss. With my own code to blame I can easily dig into what is going wrong.

[Siwastaja]

I do this:

Link two copies of application, linked at different memory halves (app_a, app_b; only linker script is different).
Calculate CRCs over app_a, app_b.
Application itself contains code to update the other half.
In the meta section, with CRC, there is also a counter value for each app.
When app_a is updating app_b, it looks at its own counter value and increments it for the other, i.e. app_b.counter = app_a.counter + 1.
Bootloader is a simple piece of code which chooses to boot the firmware which has higher counter value (unless the CRC check fails, in which case it boots the other, of course).

For the update protocol itself, either make it output a status message about which app is running, so that you can supply the other one; or simply always supply both apps back-to-back so that firmware can choose which to write (and ignore the other).

Zero problems, zero bricked devices. The only real risk is accidentally making a realease so broken that even the updater does not work. You can mitigate this by adding a pushbutton, DIP switch or something like this which just swaps the logic in bootloader, so that it boots the older one, but I haven't bothered.


The nasty thing in flash is, erase-write cycle needs enough energy that it's impractical to use e.g. capacitors to store it, so power loss during erase is always a risk. This is why there is no way around effective halving of available flash if you want zero-risk update.

[DavidAlfa]

Use the hardware CRC, it will save a lot of CPU power and make things easier.
You could use A/B partitioning, and a small bootloader that checks which is the active partition.
A small sector would be used to store the active partition ID, plus their lengths and checksums.
The bootloader should check the checksum and only boot if correct, trying to boot the other partition if wrong (Also perform checksum).
The update routine itself should be in the bootloader.
When the system receives a update request, write a magic number to certain RAM variable and reboot to enter the bootloader.
That variable should be unitialized to keep it's value after a reset.
The bootloader should check this memory address and enter update mode when set, or when both partitions have bad checksums.

[Siwastaja]

Use the hardware CRC, it will save a lot of CPU power and make things easier.

"Lot of power"? I mean, how much is that? A typical mid-grade microcontroller uses like 100mW, and most projects are not battery powered and thus do not utilize sleep modes anyway so there is absolutely no difference in power use, quite fact it would be the opposite, with hardware accelerator enabled power would go up unless you actually sleep the CPU during the time, configure it to generate event and so on. Easier?

Or maybe you meant something entirely different? Maybe boot time? Maybe energy?

Even in battery-powered products, I don't care. CRC is checked once at boot, and if it consumes some 0.0001% of battery SoC doing that, fine. Totally irrelevant because it's not a repeating consumption. Development time of copy-pasting the software CRC function (exact same code which generates the CRC on the flasher side) vs. finding out how to enable and use the HW accelerator would be in hours. Don't overoptimize trivial things.

And time used to calculate CRC is less than time spent waiting for crystal stabilization and PLL lock and whatever. Totally irrelevant. If I need some microsecond-scale IO operation at boot, I make it a special feature of the bootloader (so that it also happens before bss zeroing / .data copy). And if software CRC taking a millisecond is too slow, HW accelerated version taking half a millisecond (it's still flash bandwidth constrained) is probably too long, too.

[DavidAlfa]

Not in energy power, but in processing powering.
Hardware CRC is plain easy to use, simply feed data and get the hash.
I mean, most stm32 have it, why do it in software?

[Siwastaja]

Not in energy power, but in processing powering.

What are energy power and processing powering, never heard such terms and they make no sense in the context. Sorry the rudeness but is this drug-related?

[DavidAlfa]

Somehow the phone keyboard autocorrected power to powering.
Unless you intentionally wanted not to, I think you perfectly understood it, software CRC computing is a lot more intensive task than using the CRC module.

[Siwastaja]

I still don't get it, really. How do you measure this "intensity" and why do you think it matters?

You would be waiting in a busy loop for the HW accelerator to finish, consuming more power, maybe for a little bit less time, and could maybe increase the boot time from 2 milliseconds to 1 millisecond or something like that. Or you could micro-optimize, use sleep modes and save some microwatt-hours of energy plus a millisecond of boot time. But why do you think it is relevant at all? There are much better ways to spend your time, usually.

I think you have totally missed what we are discussing. We are discussing a bootloader which does one-time check, during which we have nothing else to do but to wait for the result. We are not discussing concurrent processing of a large dataset in the application.

[DavidAlfa]

From all the crap I wrote you took the less important part LOL.
Whatever!
+1 to the ignored list.

[nctnico]

If we're talking about the update over the UART, then please note that all STM32 have a built-in bootloader capable of updating.

Just reset the MCU in the bootloader mode (typically, pull boot0 high then reset), and use a standard STM32 bootloader protocol, described in AN3155.

That is how I let more tech savvy customers do software updates as well but for the NXP LCP series microcontrollers together with a standard UART-USB product that acts as a programming interface.

However, for a more universal update method through more convential means like a dedicated update tool, I use encrypted firmware images which also have an SHA1 or SHA256 checksum to make sure the image isn't corrupt. The advantage of SHA1 and even more so SHA256 is that (when seeded with a secret pass-phrase) they are very hard to reverse so your firmware can not be tampered with.

[paulca]

Just popping in.

My STM32 dev board, which I just blew the dust off for a project happens to have an ESP32 on the PCB with the F411.

It's been on my "TODO" list to create an ESPHome component that will flash the STM32 over the air.

I have not even started the research properly yet.  I mean I can do the ESPHome side, but haven't even looked at the STM32 side.

I was hoping to start with someone's basic UART flash bootloader.  Those things people create for their STM32s so they can program them with the arduino IDE or with PlatformIO when they simply haven't spent the $2.99 for an STLink  /mini-rant.

Those bootloaders however should give me exactly what I need, unless they include the arduino core bootloader which won't be.  I just need a bootloader that works like the arduinos.  aka it waits for a time for the UART to connect for flashing and if it doesn't it boots the existing code.

The ESP32 component will receive the incoming binary flash stream, reset the STM32, connect to the UART, flash it and reset it.

[NorthGuy]

Link two copies of application, linked at different memory halves (app_a, app_b; only linker script is different).
Calculate CRCs over app_a, app_b.
Application itself contains code to update the other half.
In the meta section, with CRC, there is also a counter value for each app.
When app_a is updating app_b, it looks at its own counter value and increments it for the other, i.e. app_b.counter = app_a.counter + 1.
Bootloader is a simple piece of code which chooses to boot the firmware which has higher counter value (unless the CRC check fails, in which case it boots the other, of course).

Microchip has this mechanism built-in in their high end PICs. The flash is divided in two partitions. They can be swapped. Config bits select which one boots. Before boot, the one being selected is mapped to lower addresses, while the other one is mapped to the higher addresses. This way, you don't even need different linker scripts.

[NorthGuy]

The ESP32 component will receive the incoming binary flash stream, reset the STM32, connect to the UART, flash it and reset it.

Could've done this with SWD as well.

[paulca]

The ESP32 component will receive the incoming binary flash stream, reset the STM32, connect to the UART, flash it and reset it.

Could've done this with SWD as well.

From the network?

[ataradov]

You can do it directly from the network if STM32 is not required for ESP32 operation.

But even with your scheme, you can substitute UART for SWD keeping everything else the same and get rid of one more component (bootloader).

[NorthGuy]

The ESP32 component will receive the incoming binary flash stream, reset the STM32, connect to the UART, flash it and reset it.

Could've done this with SWD as well.

From the network?

From ESP32.

[peter-h]

I did all this and posted sourcecode here. Search my posts and for stuff like position independent code (which I didn't actually use).

I used bits of the HAL code as examples, and loaded the flashing code into RAM.

And yes a temp copy of the new flash content has to be stored somewhere. I am using an Adesto SPI flash device for that.

I also keep a "boot block" (32k) which is not reflashed, and this prevents bricking. Actually my product does support rewriting the boot block too; there is a "bricking time window" of only about 300ms.

[tellurium]

The ESP32 component will receive the incoming binary flash stream, reset the STM32, connect to the UART, flash it and reset it.

Oh, nice, we did the same years ago, with different target MCUs.
For example, with SAMD51:

[SiliconWizard]

So the SAMD51 was just for flashing the ESP32? Wasn't it actually more expensive than the ESP32 itself?

[n4te]

Using a bootloader to swap between the first and second halves of flash is much better than my RAM copy function. I'll do that in the future!

The discussion has been interesting and entertaining. 

[ataradov]

How it is better?

On SAM D51 it is easier because you have a single bit bank swap. But most MCUs don't have that and you still have to do the work manually. And if you are doing it manually, you can do the swap in the RAM function. It won't be atomic, so you still will get a brick if power interrupts.

[n4te]

How it is better?

It's better than my copy because if the copy fails, the device is bricked until flashed via SWD. Having a bootloader that boots from the first or second half of flash leaves the other half intact. If anything goes wrong, it could boot from the other half and likely still have a device that works.

[ataradov]

If you want to have two images, then you would have to build the application twice for different halves of the flash (assuming no bank switching available). Position independent code does not really work, so you need to different images built for two separate offsets.

Then you can receive the complimentary image over the air and the "bootloader" does not need to copy anything, it just needs to pick the most recently updated half and boot it.

This is also a very common approach in wireless systems.

[SiliconWizard]

How it is better?

It's better than my copy because if the copy fails, the device is bricked until flashed via SWD. Having a bootloader that boots from the first or second half of flash leaves the other half intact. If anything goes wrong, it could boot from the other half and likely still have a device that works.

Yes. Only at the cost of having only half the flash available. Your choice.

[tellurium]

So the SAMD51 was just for flashing the ESP32? Wasn't it actually more expensive than the ESP32 itself?

Nope, vice versa.
ESP32 flashes SAMD51

ESP32 connects to a cloud server, and thus is accessible remotely.
As a result, we got a remotely controllable, update-able SAMD51

[paulca]

The ESP32 component will receive the incoming binary flash stream, reset the STM32, connect to the UART, flash it and reset it.

Could've done this with SWD as well.

From the network?

From ESP32.

Does a viable ESP32 SWD client exists? 

[ataradov]

The SWD part is easy, if you can toggle a couple pins, any MCU can do it. The part that would need work is the actual ST flash programming, which is just following ST datasheet.

Example portable SWD code is here https://github.com/ataradov/embedded-swd . The target code there for SAM D21, so that would have to be adjusted for ST.

[SiliconWizard]

So the SAMD51 was just for flashing the ESP32? Wasn't it actually more expensive than the ESP32 itself?

Nope, vice versa.
ESP32 flashes SAMD51

ESP32 connects to a cloud server, and thus is accessible remotely.
As a result, we got a remotely controllable, update-able SAMD51

Oh ok, I see.

[tellurium]

Does a viable ESP32 SWD client exists?

we have https://vcon.io - that's a closed source pre-built firmware for esp32/esp32c3, which can reflash some MCUs (including all STM32)

STM32 reflashing can be over SWD or UART.
If SWD is wired, a remote GDB debugging is possible  - https://vcon.io/docs/#live-debug-with-gdb
Remote means remote, e.g. your device could be deployed to the customer's premises, and you can debug it from your office.