How does UART bootloader program flash application data into Flash/ROM memory

[EVblog1]

I know UART bootloader flashes application HEX data into ROM/Flash but I don't understand how it flash application into ROM memory

Imagine an MCU with ROM/Flash, RAM, and a UART. Normally, I can write a C program that receives data from a serial terminal. The compiler converts the C code into a HEX file, and I use a programmer to load that HEX file into the MCU's ROM/Flash. That part is clear to me.

Now suppose I have another C program that simply blinks an LED. I have its HEX file, and I want my UART program to receive that HEX file through the serial terminal and store the application program into the MCU's ROM/Flash. In this case, the UART program is basically acting as the bootloader, and the LED-blink program is the application.

My confusion is about what happens after the UART receives the data. In a normal UART program, I understand that the received data can be stored in RAM. But in the bootloader case, the UART is also receiving the HEX data from the serial terminal. So how does the bootloader take that received HEX data and actually program the application into the MCU's ROM/Flash?

In other words, what is the actual mechanism that allows the bootloader to receive the application data through UART and then write that data into Flash/ROM?

[Siwastaja]

By following the process described in the datasheet: writing the right flash access registers to erase pages and write any data. That varies between manufacturers and product families, but the general pattern is the same: unlock the flash with some special magic number write, command page erasure and enable write access by writing to a control register in the FLASH peripheral, then write to the flash (special commands through registers, or just memory mapped).

How the bootloader deals with UART communication, buffering it, integrity checking etc. is up to the bootloader designer. The tool that produces the data for the bootloader (e.g., writes it to the UART) must be designed in tandem with the bootloader itself.

[brucehoult]

One fun part is that if the bootloader it itself in flash with XIP, you generally need to turn that off in order to write the flash, which means you first need to copy the important bits of program into SRAM and run them from there.

Doesn't apply to very low end Harvard architecture CPUs such as PIC, AVR, or 8051 which can not run program code from RAM.

[EVblog1]

So I made a high-level flowchart to explain my understanding of the process. flowchart is self-explanatory and shows how the bootloader receives the application data through UART and then writes it into Flash.

[brucehoult]

No. RAM is usually many times smaller than flash.

On my HiFive1 FE310 dev board it's 16k RAM and 16 MB flash, but that's probably an extreme case. This used almost the entire flash:

[EVblog1]

No. RAM is usually many times smaller than flash.

My point is that when the UART receives data from the serial terminal, the received data has to be stored somewhere for the software to process it. As I understand it, this is normally handled through RAM, not directly stored in ROM/Flash ( PIC, or 8051)

[Siwastaja]

I have done flashers that copy the whole thing in RAM and write it in one go after verification, but that's a special case more than a rule. Usually you would receive a piece, write and verify the piece, receive another piece and so on, because the firmware image is larger than the RAM. The piece could be as small as the smallest allowed flash write size - depends on the microcontroller, but any sensible design would do a larger chunk (maybe a few hundred bytes) and calculate and verify a checksum over it before writing.

Bootloader being fixed means that if the write fails, app does not boot but you can retry the update later. That's a relatively safe design, although it doesn't allow automatic over-the-air updates, but rather needs a UART cable, special update software, and booting the device at the right instant to start the update.

[brucehoult]

No. RAM is usually many times smaller than flash.

My point is that when the UART receives data from the serial terminal, the received data has to be stored somewhere for the software to process it. As I understand it, this is normally handled through RAM, not directly stored in ROM/Flash ( PIC, or 8051)

Yes but you don't loop back to "Bootloader start" or "Firmware update request"!

[NorthGuy]

No. RAM is usually many times smaller than flash.

My point is that when the UART receives data from the serial terminal, the received data has to be stored somewhere for the software to process it. As I understand it, this is normally handled through RAM, not directly stored in ROM/Flash ( PIC, or 8051)

Bootloader will certainly use RAM, but since there won't be enough to store the whole app image, the bootloader will have to erase the old app in the process. If something goes wrong in this process, there will be no working app, so the bootloader must recognize such situations and avoid running the app if it is not fully programmed.

To avoid this situation, it is possible to have two (or more) different places where the app can live, often called partitions. At boot, it is decided which of the partitions should run. If there are two partitions A and B, the working app may be in A, while the bootloader programs new application into B. Once it verifies that everything is written ok, it marks B as active - it will now run when the chip boots. Next time, the active app will be in B, and bootloader will re-write A, then swap them again.

By using position independent code, you can have multiple apps written in flash, and the startup logic will decide which one to run depending on circumstances. Thus, you can revert to an older version, or run an entirely different app.

[EVblog1]

often called partitions.

So, we actually partition the Flash memory into different regions for the bootloader, Application  by modifying the linker script

[NorthGuy]

often called partitions.

So, we actually partition the Flash memory into different regions for the bootloader, Application  by modifying the linker script

That's not what I meant. But yes, you can do that. You would need to  take care of interrupt vectors and such.

You can also embed a bootloader into the app. This way, your bootloader gets updated when you update the app, giving you opportunities to add new features to the bootloader too.

[Siwastaja]

Two-partition approach also means you don't need to program the flash from the bootloader; you can program from the other app. It is 100% bulletproof and easy to prove as such. The working app never erases itself, so it's always bootable.

My implementation on this is: clear the other partition, write it at any pace you wish, as the last thing write checksum and a "boot counter" which is currently running partition's counter plus 1. After that is done, reset. Then the bootloader logic is simply, "boot the app whose checksum matches the content, and if both are bootable, boot the one with higher counter".

In this approach, bootloader is tiny and trivial, but there are two full set of applications, so available flash effectively halves. This makes sense when the update process is not small or trivial; e.g. it contains full TLS networking stack, which the application uses anyway, so on the app side, the flashing is trivially simple; in the bootloader, it should replicate the whole networking stack. You could do that, and let the application use bootloader's networking stack (you can share the flash, why not), but then you need a way to upgrade that (because it is complex enough it cannot be proven 100% future-proof before the first release), and this just pushes the problem elsewhere: the only 100% reliable way is to duplicate it anyway.

The erase cycle in flashing is relatively long and power consuming. You can't realistically protect it against power loss. And if the whole firmware doesn't fit in RAM at once (as it usually doesn't), you are at the mercy of the upstream system feeding the firmware into your product - that can stall indefinitely or stop working altogether. If you cleared your only working app, any "I have a li-ion battery" doesn't help. So, if you care about product not getting bricked, dual apps or a fully independent bootloader which can do the retry.

Sometimes you will see magical thinking happening: claims of easy wins without duplicating everything, but usually there is a hole somewhere, or some amount of risk. Full duplication of the the whole app is the only obviously 100% bullet-proof when the flashing stack is large. Some people find that adding a physically separate flash chip is somehow easier for them to understand, but I find that weird. Partitioning the internal flash is always the simplest, and external chip makes sense when the otherwise good-fit MCU is already maxed out, and a larger-flash model series is significantly more expensive, complicated or weird than adding the external chip. Rarely so, unless the application is huge (e.g., high resolution graphics for UI).

[EVblog1]

Northguy and Siwastaja, have you guys actually developed a bootloader yourself, or did you use a ready-made one?

If you developed one yourself, could you share what platform you used, MCU, compiler, and communication interface and how it was working?

[Siwastaja]

Northguy and Siwastaja, have you guys actually developed a bootloader yourself,

The bootloader I describe in my above post, yes, "developed" it myself. The whole thing is 248 lines of code and that contains comments, commented-out UART debug prints, delay functions to blink a LED, some small extra application-specific features.

Calculating a CRC, comparing it, looking at a counter, and "starting the app" - the full simple bootloader which doesn't flash but only boots, as I describe above, is 30 lines of code.

I'll give you the hard part here because really, that's something everyone copypastes anyway.

Code: [Select]

static void __attribute__((naked)) start_app(uint32_t pc, uint32_t sp)
{
__asm(
"msr msp, r1\n\
bx r0\n");
}

static void __attribute__((noreturn)) boot(void* addr)
{
SCB->VTOR = (uint32_t)addr;
uint32_t sp = ((uint32_t*)addr)[0];
uint32_t pc = ((uint32_t*)addr)[1];
start_app(pc, sp);
while(1);
}


[ejeffrey]

Northguy and Siwastaja, have you guys actually developed a bootloader yourself, or did you use a ready-made one?

If you developed one yourself, could you share what platform you used, MCU, compiler, and communication interface and how it was working?

I developed one for the NXP Kinetis M0+ processors.  It has two versions, one that uses a serial port, and one that uses a wiznet SPI ethernet chip as the primary communication device.  The work essentially identically, it's just a question of whether read() reads from a TCP socket or a UART.

I used two partitions set up by the linker script.  The bootloader and the main program are completely separate projects.

The bootloader is designed to never be updated.  It won't write to it's own partition and the main program has no flash write code at all.  So you have to get out a SWD probe if you want to modify it.  At power up, the bootloader checks for a hardware button being pressed, a magic command on the serial port within the first 2 seconds, or a corrupt first block (the interrupt table) for the application partition, and if finding any of them enters bootloader mode, otherwise it chains to the main proram.

The bootloader update process is:

1) erase the first page of the application program.
2) Writes the application code in blocks of 1024 bytes, in reverse order from the end of the flast to the beginning.
3) The final block written is block 0, the interrupt vector block
4) reset

The idea is that after step 1 is complete, if the process is interrupted, it will be detected at power up and automatically sit in firmware update mode.  This was just a lazier way to avoid implementing firmware signing or checksums.

The main program project itself has a stub loader at flash address zero that unconditionally chains to the main program.  The idea here is that if I flash the main program from the SWD dongle (say for debugging) it overwrites the bootloader and runs as a stand-alone process.  But since this stub is in the bootloader partition, it gets dropped from the firmware update.

The bootloader code itself is a couple hundred lines of code, but in my case I also included a bunch of code from the main project that does hardware diagnostics, and for the ethernet version device discovery.  So the bootloader is actually fairly large in terms of flash usage, but that wasn't a constraint for my project.

I have a small python script that merges the IntelHex files from the bootloader and the application targets to create the initial factory programming image.  That way I get a default version of the application immediately.

[NorthGuy]

Northguy and Siwastaja, have you guys actually developed a bootloader yourself, or did you use a ready-made one?

Long time ago. I remember the last one was very simple, much simpler than Siwastaja describes. It simply executed read/erase/write commands from the controlling app and had some basic safety (like it woudn't overwrite the working app or the bootloader itself). The controlling app was run on PC, was doing all the work, and sending commands to the bootloader. But then I thought that instead of using a converter chip, I could use a small MCU, such as PIC16, which would re-program the target chip and then I wouldn't need a bootloader any more ...

[ejeffrey]

To add:

In my application, it was deemed acceptable for a corrupt flash write to leave the device non-functional as long as it can be recovered.  So we didn't store two full copies of the application image like Siwastaja describes.  If a flash fails, we don't have a known good image to fall back to, we just sit in the bootloader loop until we can recover by re-running the flash update process.

The bootloader could have bugs in it, since it is considerably more complicated and includes the communication and flash logic.  However, realistically, once it's tested to be able to program an application image and chain to it, I'm pretty confident that there are no show stoppers.  Since we have complete control over the flash update client, the fix for any bugs is going to be to document "don't do that".

The A/B image Siwastaja described is definitely a more elegant solution.  It's just wasn't needed for my application.

[brucehoult]

The A/B image Siwastaja described is definitely a more elegant solution.  It's just wasn't needed for my application.

And it requires you to spec chips with twice as much flash as you'd otherwise need.