goal is to develop custom bootloaders for AVR DA/DB MCUs, probably ATmega4809 too. My development environment is MPLAB X IDE with the XC8 compiler.
XC8 is a brand name which ... since they bought Atmel it has the added avr-gcc for AVR.
So, in short, you're using avr-gcc.
Unfortunately, if you set the compiler to XC8, microchip wraps their own code around the linker and compiler, and the "options" that you need to set for bootloader use move, disappear, or don't work. :-(
Note that there are some pretty significant differences between the NVMCTRL stuff between the AVR-xxx chips and the ATmega4809 and "xTiny" chips.
Let's start with the bootloader code:
1. How do I tell the toolchain that this program should reside in the BOOT section? Is there a setting I can use in the bootloader's C code itself?
For these newer AVRs, the bootloader section starts at 0x0, which is the same as the default compile/link destination, so you don't need to do anything special.
In theory, for clarity in your source code you could put all your bootloader functions in a bootloader "section", but it's unnecessary.
2. Once the bootloader finished its job, how do I jump to the "regular" program, while making sure it will behave normally (e.g. set all registers to defaults, automatically zero-fill variables at startup)?
The usual method is to do a software reset (on newer chips) or a wdt reset (on older avrs), which sets all the IO registers and such to their defaults. Then the bootloader should detect this reset cause and do a simple jmp to the start vector in the application section (at relative address 0x0 in the application section.)
Note that zero-filling variables is part of the C startup code, which should still be present in your application.
And the main code:
1. Here too, how do I tell the toolchain that this goes to the program code segment of the FLASH? Or is that the default?
You have to move the application to the start of the application section. Normally, you'd either modify the linker script, fill in a field in the ld options, or specify an "extra link option" like "-Wl,--section-start=.text=0x1000"
Unfortunately:
- setting up an XC8 project does not copy a linker script into your project, or easily allow you to specify a custom linker script. (Maybe it's an MPLABX think rather than an XC8 thing.)
- The nice "program offset" option that exists for avr-gcc builds is not present for XC8.
- The "--section-start" option (which is what the xc8 manual says to use, BTW) doesn't seem to work right (the command line generate works fine if you repace the "xc8-ld" with "avr-gcc", so I'm not sure what's wrong.)
There is a "legacy" form "-Ttext=0x1000" that does the same thing as section-start, and that DOES seem to work, so that's what I'd try in the short term. (Maybe it's been fixed in 6.3; my experiments have been with 6.25)
2. How do I set up and refer to different interrupt vector tables for the bootloader vs. the main code?
The vectors get relocated along with the .text segment, so you don't need anything further to adjust the addresses of ISRs. The vector table location defaults to the beginning of the application section, so if your bootloader wants to use interrupts, it MUST change IVSEL in its startup code (before enabling interrupts.)
There are several examples you can look at:
https://github.com/Optiboot/optiboot/blob/master/optiboot/bootloaders/optiboot/optiboot_x.c bootloader for ATmega4809 and xTiny.
https://github.com/SpenceKonde/DxCore/tree/master/megaavr/bootloaders/optiboot_dx bootloader for AVR-D*
(These are both based on "optiboot", but optiboot_x is more of a re-write, and optiboot_dx is more of a modification. Because of the differences in NVMCTRL, I think.)