How does UART bootloader program flash application data into Flash/ROM memory

[cv007]

I bet many users would be more willing to install your custom application, than to use some modem-era software they have never heard about

That custom application can just as easily deal with the mcu that is using an xmodem bootloader, so one can have both- custom application for those that need/want it and can also use xmodem transfer (and any other needed interaction) via a terminal app like tera term for any others that can handle that.

I have a bootloader in the online compiler, for a modern avr (xmodem-crc)-
https://godbolt.org/z/YMWczGTrx

I have written various bootloaders starting with early pic's, and the above was simply so I could use sx on a linux pc and also tera term on a Windows pc. I previously wrote avr bootloaders that could take in a hex file, where I could drag/drop the hex file on tera term but that required adding a line delay as there is no flow control when doing something like that. With the above xmodem-crc bootloader I do convert to binary in the build so I do not need translation in the mcu.

Normally I have little need for bootloaders anymore, but for a current project I have a single wire coming out of a box for uart debug info and if I need to reprogram I would rather not open the box to get at the programming pins (although in this avr there is a single updi programming pin which I can tie to the one-wire uart pin and make it work, just not willing to tie the updi pin to the uart pin even though the probability of inadvertently entering programming is pretty slim). With this setup I can remain in tera term while watching debug info and do any reprogramming if needed.

I also have a bootloader for the newer avr where the bootloader has a simple protocol and very little brains (basically read n bytes from addr, or write n bytes to addr). The pc software does all the work, including writing to the appropriate nvm registers to do the flash write. So you basically have read/write access to the ram/peripherals from the pc. Its simple on the mcu side, but you just moved the work to the pc side and probably end up with more work in the end.

[abyrvalg]

Another kind of bootloader (not for true Harvard CPUs like PIC/AVR/8051): just two commands - “write to RAM” and “jump to RAM”. The real job is performed by the external code loaded to RAM. This allows to define the actual functionality later. I.e. you can decide to do things like reformatting some settings area to a newer structure together with app update, or read something out for fault analysis, or perform some kind of init/test/calibration, or any combination of these things by loading a specific “RAM loader”, performing the function, rebooting/returning back to bootloader, loading another one.

[NorthGuy]

I also have a bootloader for the newer avr where the bootloader has a simple protocol and very little brains (basically read n bytes from addr, or write n bytes to addr). The pc software does all the work, including writing to the appropriate nvm registers to do the flash write. So you basically have read/write access to the ram/peripherals from the pc. Its simple on the mcu side, but you just moved the work to the pc side and probably end up with more work in the end.

This is what SWD can do on ARM32 directly, along with nearly anything else, Doesn't require any bootloader at all.

[eutectique]

Do you have a specific standard / field in mind where this is a requirement?

The EU Cyber Resillience Act which has a broad scope (including industrial IoT). From December 2027 it is part of CE compliance.

You might be interested in a webinar hosted by Memfault named "Shipping Firmware Under the CRA: Nicolas Schieli on the Questions Everyone’s Asking, and the Ones They Should Be"

https://memfault.com/resources/coredump-024-shipping-firmware-under-the-cra-nicolas-schieli/

It starts in 25 minutes, sorry for the late link, I just happen to come across new comments in this topic.

[cv007]

This is what SWD can do on ARM32 directly, along with nearly anything else, Doesn't require any bootloader at all.

The newer avr has updi which is similar with a single wire using uart half-duplex protocol. Certainly no need for a bootloader if you can make use of native programming.

[westfw]

Certainly no need for a bootloader if you can make use of native programming.

UPDI, and some other built-in programming capabilities, tend to be on pins that are NOT the same as a product might use to communicate under normal circumstances (eg USB or UART.)  So you frequently see (for example) UDPI-capable AVRs with an added UART bootloader, or STM32F103 boards ("Blue Pill") with a USB bootloader...

[eutectique]

You might be interested in a webinar hosted by Memfault named "Shipping Firmware Under the CRA: Nicolas Schieli on the Questions Everyone’s Asking, and the Ones They Should Be"

For anyone interested in the effects of CRA coming into force in September 2026, here is the webinar recording:

[peter-h]

I started a thread here on the EU security thing. I don't think many people have time to watch a 1hr video
https://www.eevblog.com/forum/microcontrollers/eu-cyber-regulation-and-secure-firmware-updates-mandatory/

Re boot loaders, lots of ways to do it and with a lot of history.

One of the oldest formats is intex hex. You have a line of about 5 bytes, in ascii-hex, 2 bytes per binary byte, and each line starts with the load address. So you receive this line into RAM, check the checksum on it, and program it into FLASH. It usually relies on FLASH being much faster to program than the data is arriving, which is usuall ok.

The boot loaders in STM arm32 chips just receive data and program each byte into flash. Not sure there is any addressing. The loader code runs in a different address space to the flash being programmed; this is obviously important since you cannot normally execute from the same flash bank (there are exceptions to this and e.g. the 32F4 can, in certain ways). I wrote a RAM-resident loader to do it properly - see threads on eevblog on loaders, relocatable code, etc.

[MIS42N]

I started a thread here on the EU security thing. I don't think many people have time to watch a 1hr video
https://www.eevblog.com/forum/microcontrollers/eu-cyber-regulation-and-secure-firmware-updates-mandatory/

You are right. TL:DR (well not Read but Watch - I felt sleepy after a minute). So I looked up EU CRA and there's a Wikipedia article. It seems open source is exempt so I am in the clear. I'm happy for people to hack anything I make public.

Re boot loaders, lots of ways to do it and with a lot of history.

Every computer has to have a boot loader. Historically it was in a ROM, the CPU would fetch its first instruction from a fixed address and proceed from there. The instructions may initiate a primitive user interface allowing the user to select a device from which to load a program. Many recent microprocessors have a programming state (for example raise a pin to a specific voltage) which activates something in the processor which allows it to be programmed. So the examples in this thread are really secondary loaders, originally loaded into the processor by the hardware loader. In the PC world the primary loader is the BIOS. For early machines it usually loads IIRC 128 bytes from a selected device, for a disk or USB drive this was from sector 0. The second 128 bytes had a partition table. The 128 bytes was a secondary loader that started loading another loader from the disk (sectors 1-?) which was a tertiary loader that was responsible for loading the operating system. I'm not up to date on the latest PC load system.

 
One of the oldest formats is intex hex. You have a line of about 5 bytes, in ascii-hex, 2 bytes per binary byte, and each line starts with the load address. So you receive this line into RAM, check the checksum on it, and program it into FLASH. It usually relies on FLASH being much faster to program than the data is arriving, which is usuall ok.

Intel hex is a little more complex than that. The loader I wrote for the PIC reads Intel format, as that is the output of the tool chain. A record starts with : followed by a byte count and ends with a checksum. The data in between can be a few different things. There can be padding bits between a checksum and the next : to allow time for programming to take place. I didn't need that because XMODEM implementation programs the flash before sending an ACK.

[peter-h]

Yes of course. Typo: I meant 50 bytes. Probably 64 mostly. But the key point is that each intel hex line is standalone and gets programmed at the stated address. There can be gaps...

I wrote intel hex loaders in the old days.

[brucehoult]

each intel hex line is standalone and gets programmed at the stated address.

They're not standalone. There is a current base address to which the address in each line needs to be added. Original HEX was 16 bit addresses, then they extended it for segmented 20 bit, and later 32 bit. There is no standard 64 bit extension.

I wrote intel hex loaders in the old days.

I wrote a simple and overly-permissive one a couple of years ago.

https://github.com/brucehoult/trv/blob/main/trv.c#L55

[voltsandjolts]

I guess for under 64K payload the base address is 0 (assumed for mcu fw) and you can read each line stand-alone.

[brucehoult]

I guess for under 64K payload the base address is 0 (assumed for mcu fw) and you can read each line stand-alone.

Lots of MCUs have more than 64k of flash. Even the 8 bit 16 MHz ATmega2560 from 2007 has 256k.

[voltsandjolts]

Yes, of course they do these days. I was just pointing out that Peter wasn't wrong, just an outdated strategy.

[gspeed]

Two-partition approach also means you don't need to program the flash from the bootloader; you can program from the other app. It is 100% bulletproof and easy to prove as such. The working app never erases itself, so it's always bootable.

My implementation on this is: clear the other partition, write it at any pace you wish, as the last thing write checksum and a "boot counter" which is currently running partition's counter plus 1. After that is done, reset. Then the bootloader logic is simply, "boot the app whose checksum matches the content, and if both are bootable, boot the one with higher counter".

In this approach, bootloader is tiny and trivial, but there are two full set of applications, so available flash effectively halves. This makes sense when the update process is not small or trivial; e.g. it contains full TLS networking stack, which the application uses anyway, so on the app side, the flashing is trivially simple; in the bootloader, it should replicate the whole networking stack. You could do that, and let the application use bootloader's networking stack (you can share the flash, why not), but then you need a way to upgrade that (because it is complex enough it cannot be proven 100% future-proof before the first release), and this just pushes the problem elsewhere: the only 100% reliable way is to duplicate it anyway.

The erase cycle in flashing is relatively long and power consuming. You can't realistically protect it against power loss. And if the whole firmware doesn't fit in RAM at once (as it usually doesn't), you are at the mercy of the upstream system feeding the firmware into your product - that can stall indefinitely or stop working altogether. If you cleared your only working app, any "I have a li-ion battery" doesn't help. So, if you care about product not getting bricked, dual apps or a fully independent bootloader which can do the retry.

Sometimes you will see magical thinking happening: claims of easy wins without duplicating everything, but usually there is a hole somewhere, or some amount of risk. Full duplication of the the whole app is the only obviously 100% bullet-proof when the flashing stack is large. Some people find that adding a physically separate flash chip is somehow easier for them to understand, but I find that weird. Partitioning the internal flash is always the simplest, and external chip makes sense when the otherwise good-fit MCU is already maxed out, and a larger-flash model series is significantly more expensive, complicated or weird than adding the external chip. Rarely so, unless the application is huge (e.g., high resolution graphics for UI).

I wrote a quite similar one, except that the bootloader always checks the second partition first, and if there's a valid image, it always copies that to the first partition and then erase the second partition. So you get atomic and consistent updates, while the program always executes from the same position (easier to build/debug).

[Siwastaja]

Two-partition approach also means you don't need to program the flash from the bootloader; you can program from the other app. It is 100% bulletproof and easy to prove as such. The working app never erases itself, so it's always bootable.

My implementation on this is: clear the other partition, write it at any pace you wish, as the last thing write checksum and a "boot counter" which is currently running partition's counter plus 1. After that is done, reset. Then the bootloader logic is simply, "boot the app whose checksum matches the content, and if both are bootable, boot the one with higher counter".

In this approach, bootloader is tiny and trivial, but there are two full set of applications, so available flash effectively halves. This makes sense when the update process is not small or trivial; e.g. it contains full TLS networking stack, which the application uses anyway, so on the app side, the flashing is trivially simple; in the bootloader, it should replicate the whole networking stack. You could do that, and let the application use bootloader's networking stack (you can share the flash, why not), but then you need a way to upgrade that (because it is complex enough it cannot be proven 100% future-proof before the first release), and this just pushes the problem elsewhere: the only 100% reliable way is to duplicate it anyway.

The erase cycle in flashing is relatively long and power consuming. You can't realistically protect it against power loss. And if the whole firmware doesn't fit in RAM at once (as it usually doesn't), you are at the mercy of the upstream system feeding the firmware into your product - that can stall indefinitely or stop working altogether. If you cleared your only working app, any "I have a li-ion battery" doesn't help. So, if you care about product not getting bricked, dual apps or a fully independent bootloader which can do the retry.

Sometimes you will see magical thinking happening: claims of easy wins without duplicating everything, but usually there is a hole somewhere, or some amount of risk. Full duplication of the the whole app is the only obviously 100% bullet-proof when the flashing stack is large. Some people find that adding a physically separate flash chip is somehow easier for them to understand, but I find that weird. Partitioning the internal flash is always the simplest, and external chip makes sense when the otherwise good-fit MCU is already maxed out, and a larger-flash model series is significantly more expensive, complicated or weird than adding the external chip. Rarely so, unless the application is huge (e.g., high resolution graphics for UI).

I wrote a quite similar one, except that the bootloader always checks the second partition first, and if there's a valid image, it always copies that to the first partition and then erase the second partition. So you get atomic and consistent updates, while the program always executes from the same position (easier to build/debug).

Updates are as "consistent" and "atomic" either way, so really the only difference is, you don't have to link two copies of the application, and either supply them both during update (and have the firmware ignore one of them), or communicate which version is desired (this is what we do - there is continuous data flow from the device anyway, so whether it's now running "A" or "B" image is a single bit, so server can choose whether to send B or A for the update). And linking two versions is just two linker scripts and two calls to the linker. Another option would be position-independent code but that would have some downsides (slightly bigger code for example).

Your way has a downside that the boot-after-update is slower, but usually that would not matter. Also flash endurance is halved compared to my solution since every update involves two erases and writes. Doesn't matter either, so really either way is fine and it's up to preference / whichever you happened to choose originally.

[peter-h]

One guy I've met (not on here AFAIK) did a design with a 2nd (simple) CPU which programs the FLASH in the main CPU, saying that this was the only way to provably make the product brick-safe.

I doubt that is actually correct, so there may have been something else...

Surely a product is brick-safe if there is a "boot block" which is never reprogrammed. That is how I've done my 32f4xx stuff. Bottom 32k is not touched (except, in general, when working with a debugger, when the entire ELF file gets written, obviously). There is a way to OTA write the whole lot and then you have a "brick window" which is just the programming time of that 32k (actually two 16k blocks but since the code is close to 16k I used 32k), which IIRC is around 300ms. The running code is in RAM and will retry a few times. You would need to be extraordinarily unlucky.

[Siwastaja]

One guy I've met (not on here AFAIK) did a design with a 2nd (simple) CPU which programs the FLASH in the main CPU, saying that this was the only way to provably make the product brick-safe.

Of course not. Both my and gspeed's solutions are provably brick safe. They are very simple to reason about and prove: the application never ever writes over itself, or over its boot metadata, or the bootloader. As such, the running application always remains bootable: all parts that participate in booting the app, and the app itself, are untouched. Q.E.D., end of story. The cost is needing flash for two copies. That is what buys the non-brickage. The advantage in what I described is the absurdly simple bootloader which is so small that it's easy to reason about and prove good.

A multi-MCU solution for something which doesn't need it is a surefire way to increase complexity, and generally adding unnecessary complexity in something where you want to prove reliability isn't a very good idea.

[nctnico]

Two-partition approach also means you don't need to program the flash from the bootloader; you can program from the other app. It is 100% bulletproof and easy to prove as such. The working app never erases itself, so it's always bootable.

My implementation on this is: clear the other partition, write it at any pace you wish, as the last thing write checksum and a "boot counter" which is currently running partition's counter plus 1. After that is done, reset. Then the bootloader logic is simply, "boot the app whose checksum matches the content, and if both are bootable, boot the one with higher counter".

In this approach, bootloader is tiny and trivial, but there are two full set of applications, so available flash effectively halves. This makes sense when the update process is not small or trivial; e.g. it contains full TLS networking stack, which the application uses anyway, so on the app side, the flashing is trivially simple; in the bootloader, it should replicate the whole networking stack. You could do that, and let the application use bootloader's networking stack (you can share the flash, why not), but then you need a way to upgrade that (because it is complex enough it cannot be proven 100% future-proof before the first release), and this just pushes the problem elsewhere: the only 100% reliable way is to duplicate it anyway.

The erase cycle in flashing is relatively long and power consuming. You can't realistically protect it against power loss. And if the whole firmware doesn't fit in RAM at once (as it usually doesn't), you are at the mercy of the upstream system feeding the firmware into your product - that can stall indefinitely or stop working altogether. If you cleared your only working app, any "I have a li-ion battery" doesn't help. So, if you care about product not getting bricked, dual apps or a fully independent bootloader which can do the retry.

I wrote a quite similar one, except that the bootloader always checks the second partition first, and if there's a valid image, it always copies that to the first partition and then erase the second partition. So you get atomic and consistent updates, while the program always executes from the same position (easier to build/debug).

Typically I implement something similar. The uploaded firmware image goes into the 2nd half of the flash. The bootloader can then verify the image for integrity and suitability and if those check out, the image is unpacked (decrypted) and programmed into the application flash area. For a more complex system I implemented a central file server style unit (with external flash) which can distribute firmware images to various modules while supporting different hardware types and versions as well. For this the firmware images not only contain a version but also a hardware ID and version.

[Siwastaja]

Two-partition approach also means you don't need to program the flash from the bootloader; you can program from the other app. It is 100% bulletproof and easy to prove as such. The working app never erases itself, so it's always bootable.

My implementation on this is: clear the other partition, write it at any pace you wish, as the last thing write checksum and a "boot counter" which is currently running partition's counter plus 1. After that is done, reset. Then the bootloader logic is simply, "boot the app whose checksum matches the content, and if both are bootable, boot the one with higher counter".

In this approach, bootloader is tiny and trivial, but there are two full set of applications, so available flash effectively halves. This makes sense when the update process is not small or trivial; e.g. it contains full TLS networking stack, which the application uses anyway, so on the app side, the flashing is trivially simple; in the bootloader, it should replicate the whole networking stack. You could do that, and let the application use bootloader's networking stack (you can share the flash, why not), but then you need a way to upgrade that (because it is complex enough it cannot be proven 100% future-proof before the first release), and this just pushes the problem elsewhere: the only 100% reliable way is to duplicate it anyway.

The erase cycle in flashing is relatively long and power consuming. You can't realistically protect it against power loss. And if the whole firmware doesn't fit in RAM at once (as it usually doesn't), you are at the mercy of the upstream system feeding the firmware into your product - that can stall indefinitely or stop working altogether. If you cleared your only working app, any "I have a li-ion battery" doesn't help. So, if you care about product not getting bricked, dual apps or a fully independent bootloader which can do the retry.

I wrote a quite similar one, except that the bootloader always checks the second partition first, and if there's a valid image, it always copies that to the first partition and then erase the second partition. So you get atomic and consistent updates, while the program always executes from the same position (easier to build/debug).

Typically I implement something similar. The uploaded firmware image goes into the 2nd half of the flash. The bootloader can then verify the image for integrity and suitability and if those check out, the image is unpacked (decrypted) and programmed into the application flash area. For a more complex system I implemented a central file server style unit (with external flash) which can distribute firmware images to various modules while supporting different hardware types and versions as well. For this the firmware images not only contain a version but also a hardware ID and version.

The safety against brickage here comes from the fact that the validity checks are simple enough to prove obviously deterministic. If the flashing fails e.g. due to power interruption, then it simply restarts again, until it succeeds and application boots. The bootloader is slightly more complex than in my example, but still very simple and obvious.

In other words:
Anything invalid in input data -> erase never starts, old app always intact
Input data considered valid -> erase starts, but input data itself is never touched
If erase/write fails -> same input data, same outcome, retry guaranteed.

[NorthGuy]

The bootloader can then verify the image for integrity and suitability and if those check out, the image is unpacked (decrypted) and programmed into the application flash area.

How the hell a bootloader can verify stability of a packed image?

[nctnico]

The bootloader can then verify the image for integrity and suitability and if those check out, the image is unpacked (decrypted) and programmed into the application flash area.

How the hell a bootloader can verify stability of a packed image?

By reading or unpacking the headers (which are signed as well) and checking if the image is suitable for use. Keep in mind that cyphers used for encrypting data are typically block cyphers so you can do a partial decryption and read data into RAM.

[peter-h]

Not doing block chaining does reduce security but it's a pointless argument in practice for this type of application.

[nctnico]

Not doing block chaining does reduce security but it's a pointless argument in practice for this type of application.

No, you really need to use a block chain based cypher like AES-CBC to encrypt firmware (or any other data). Otherwise the encryption is pretty much useless.

[peter-h]

Not sure about useless, but the same plaintext block produces the same ciphertext block, so you can pretty much tell the size of the image, but with AES there is no known attack even if not using CBC.