Wanted: Technical advice/tips for a PIC18F Bootloader

[igendel]

Hi all,

I'm about to start my first project with a bootloader of my own in it. The MCU will be a PIC18F45K22 (or a very close one), and I'll be using MPLAB X / XC8.

The thing I'm not sure about it how to build the code so that the bootloader and the primary code will go each to its proper place. That is, technically, how do I tell the compiler that program X is a bootloader and should be written to the FLASH at position [whatever], and that the HEX file for program Y should have FLASH addresses elsewhere... is there a compiler flag? Should I edit some map file (and how do I do it)?

I searched for PIC bootloaders and app notes, but so far could only find information that's irrelevant to my specific question.

Any help, suggestions and gotcha warnings will be appreciated 

[hans]

Linker file

[igendel]

Linker file

Well, that's definitely terse
I'll explore Google, but meanwhile, if someone can help with some specific details or things to watch out for...

Thanks

[Edit: I found this http://www.microchip.com/webinars.microchip.com/WebinarDetails.aspx?dDocName=en558478 , seems to be very useful but it's 4 years old - hopefully the compiler and IDE didn't change too much since]

[JPortici]

I'm in the process of writing one myself (i also received a notification from the forum. was it you?)
provided you are using XC8, in the project settings you select XC8-Linker. Select the tab "Additional Options" and adjust offset.
the Offset parameter shift the start of the code at that particular address, leaving everything below empty.
It is different than adding a fixed number to each address of the hex file or something like that, as that would break every access at program memory like reading a constant.

So, you write your code and set Offset to be 800 or whatever (hex) address is the start of the second memory block. I'm writing one for the 26k22 which should have the same boot/page size but please refer to the datasheet of you MCU.

[JPortici]

surely you want to avoid compiler generated crap in the interrupt handling. Maybe this is not necessary (I have to investigate) but here it is:
I don't need interrupts in the bootloader as the communication will need handshacking and the PC side will wait for a redy signal from the bootloader
So my interrupt handler just need to be a goto to the application interrupt handler.. but if i write this

Code: [Select]

void __interrupt() intHroutine() {
  /* High priority interrupt routine */
  asm("GOTO 0x802");
}

the compiler will waste a lot of time saving and restsoring all the context prior actually executing the interrupt routine.. and your application will be doing that AGAIN. Being unable to disable context saving in free mode i solved the issue in this way

Code: [Select]

void __interrupt() __section("intHsect") intHroutine() {
  /* The use of section moves all the compiler crap away. interrupt vector is
   * actually a goto to the application vector.
   * add to linker command this line: "-L-pintHsect=800h" */
  NOP();
}
(cleaner ways to do so are appreciated)

[hans]

Linker file

Well, that's definitely terse
I'll explore Google, but meanwhile, if someone can help with some specific details or things to watch out for...

Thanks

[Edit: I found this http://www.microchip.com/webinars.microchip.com/WebinarDetails.aspx?dDocName=en558478 , seems to be very useful but it's 4 years old - hopefully the compiler and IDE didn't change too much since]

Sorry, didn't have that much this morning, but wanted to give at least 1 hint to what you should look out for.

In the linker file you can move which flash regions the program should be compiled to. You can limit the region for the bootloader and the application.

How you define those regions is up to you, and sometimes defined by the constraints of the chip. E.g. if the chip has bootloader FLASH, it can make sense to put the bootloader in that space.

It's ofcourse important to link and write the program to the right space in FLASH because all calls are done by absolute addresses.

Other 2 challenges are:
1) Jumping to program (usually you can figure out the main() function location from the vector table)
2) Remapping interrupts to those of the application. Interrupt handlers are referred by the interrupt vector table. For example on most ARM processors for example you can put this table into RAM and set a register in the CPU to use the RAM vector table. I'm not sure how this works on PIC18; possibly there is also a register that remaps this address correctly.

[igendel]

It actually looks easier in that Microchip video I found (see above)

I think I know enough now to start experimenting, but there's one question that occurred to me while reading your answers: what's all this about redirecting/sharing interrupts? From the video, it seems each code (bootloader/primary) has its own interrupt vector "table" (it's PIC18F, so it's just two entries). Under what circumstances would I want to enable a primary-code interrupt while the bootloader is still running?

Thanks again

[JPortici]

in the pic18 there are only two interrupt vectors (high and low priority.. but i think the priority system is bugged and i never have seen it in use anyway) that are not relocatable. and of course only one reset vector.

this means that either the bootloader reside in the lower part of memory or it resides in other parts (usually the highest memory page).
analyzing the two possibilities:

BOOTLOADER IN THE BOOT SECTOR
- MCU start and reset in bootloader.
- Interrupt vector is in the bootloader
- Ideally you don't change and even write protect the bootloader (to be sure it doesn't write itself due to a bug) so in this case the whole boot block gets locked. this means that you can't change the interrupt vector and whenever the application needs to service an interrupt it will jump back in the bootloader area. even if you don't use interrupts in the bootloader you will have to get outside and back in the application in some way
BOOTLOADER IN ANOTHER SECTOR
- The application start first
- No need to change the interrupt vector/table/location
- But entering the interrupt depends on the application. if that part is omitted or gets corrupted you're screwed. you have to re-flash everything

[igendel]

in the pic18 there are only two interrupt vectors (high and low priority.. but i think the priority system is bugged and i never have seen it in use anyway) that are not relocatable.

OK, that seems correct... in that video they indeed define a JMP from the normal vector (at the low address) to the application's vector, using some compiler magic words.
Which brings me to a different question - what if I want both the bootloader AND the application to use interrupts? Do I have to include some if in the ISR to see who is calling it, and act accordingly?

[JPortici]

exactly. for example declare a variable, common between the two, at a fixed address. if it's clear go to the application ISR otherwise skip.. or whatever.
but why would you need interrupts? you can't implement some sort of flow control?

[igendel]

but why would you need interrupts? you can't implement some sort of flow control?

I'm not sure yet, but my Bootloader may use UART as well as a primitive UI (blinking LED / Switch). It doesn't feel right to do it all with just polling 

A fixed-address RAM variable as a flag seems a bit of a hack - but it makes perfect sense. I'll try all these techniques and update this thread when I have results.

Thanks again everyone!

[JPortici]

but are you running at 4 MHz? in that case it would be understandable or if you are using higher / more complex protocols like spi/i2c. or if you don't have flow control, handshacking, .. , sure.

but ideally your bootloader should be idling and checking for new data most of the time. the only time intensive task is the actual programming but that can't be made faster. it's physics! and you don't always do that, you have to erase/program 64 bytes at a time.

[igendel]

but are you running at 4 MHz? in that case it would be understandable or if you are using higher / more complex protocols like spi/i2c. or if you don't have flow control, handshacking, .. , sure.

but ideally your bootloader should be idling and checking for new data most of the time. the only time intensive task is the actual programming but that can't be made faster. it's physics! and you don't always do that, you have to erase/program 64 bytes at a time.

Hmmm... I see now a strong argument against shared interrupts - they will create a very unhealthy dependency between the bootloader code and the application code (after all, a truly shared ISR can only be written in one of them).

However, suppose I write ISRs in both codes, but instead of just redirecting the interrupt vector I do this:

Code: [Select]

// in Bootloader code

#define APP_INT (0x300+0x8)

char isBootloader @ 0x05 = 1;

// ...

void interrupt blISR() {

   if (isBootloader) {

     // Do bootloader stuff

   } else asm("goto " ___mkstr(APP_INT));

}

Code: [Select]

// Application code

char isBootloader @ 0x05 = 0;

// ...

void interrupt appISR() {

  // Do app stuff

}

[voltsandjolts]

Maybe not applicable to your scenario but Mike discusses some interesting bootloader ideas here:



(Broken link? You Tube keywords: mikeselectricstuff bootloader)

[igendel]

Maybe not applicable to your scenario but Mike discusses some interesting bootloader ideas here:

Great video, although as you said, it's not related specifically to PIC18F bootloader technique.

[JPortici]

but are you running at 4 MHz? in that case it would be understandable or if you are using higher / more complex protocols like spi/i2c. or if you don't have flow control, handshacking, .. , sure.

but ideally your bootloader should be idling and checking for new data most of the time. the only time intensive task is the actual programming but that can't be made faster. it's physics! and you don't always do that, you have to erase/program 64 bytes at a time.

Hmmm... I see now a strong argument against shared interrupts - they will create a very unhealthy dependency between the bootloader code and the application code (after all, a truly shared ISR can only be written in one of them).

However, suppose I write ISRs in both codes, but instead of just redirecting the interrupt vector I do this:

Code: [Select]

// in Bootloader code

#define APP_INT (0x300+0x8)

char isBootloader @ 0x05 = 1;

// ...

void interrupt blISR() {

   if (isBootloader) {

     // Do bootloader stuff

   } else asm("goto " ___mkstr(APP_INT));

}

Code: [Select]

// Application code

char isBootloader @ 0x05 = 0;

// ...

void interrupt appISR() {

  // Do app stuff

}

whatever you end up doing, check the actual program memory at the interrupt vector and see if it does what you want. with the code i posted before the interrupt vector became GOTO 808h which was exactly what i wanted (you guessed it, the application interrupt vector is at address 808h)

[Ian.M]

The problem with interrupts and PIC bootloaders is preserving the context while still being able to jump to the application interrupt vector cleanly without excessive lag.  If you write the interrupt dispatcher in C, there is a lot of overhead.  It is possible to write it in assembler, and if you are careful to only use instructions that do not affect STATUS, WREG or BSR and put your flag in the Access bank, you can jump to the corrisponding vector in the main program if not in bootloader mode with minimal extra latency (only a few Tcy). 

However its far simpler and lower latency to simply place a GOTO at the interrupt vector and use a polling bootloader rather than an interrupt driven one.

See http://www.microchip.com/forums/m899466.aspx for details of how to place assembler code of your choice at the interrupt vectors cleanly.

[igendel]

That's not a bootloader, that's one deep rabbit hole... 

[JPortici]

See http://www.microchip.com/forums/m899466.aspx

As usual, thank you Ian for pointing that out. I was doing something let's say unorthodox? that is indeed a much cleaner way to do the same thing, which magically freed up 300 program words that were being allocated and counted but not compiled (because i relocated the interrupt function outside the program memory.

now my bootloader is getting to a respectable size. not that i care too much, the unused boot space is wasted anyway

[kony]

Funny thing about (some of*) the 18F series is that fusebits are actually stored at the end of device FLASH instead of its own dedicated nonvolatile memory.

This leads to dilema how to use flash write protection mechanism to protect bootloader - if memory protection is enabled starting from the start (0x0000) of the program flash, last page with CONFIGx bits is left vunerable to erase/overwrite wich might result in bricked device (and write protection configuration itself is stored there as well).
If protection from the end of memory space is used, initial reset and IRQ vectors handlers are left vulnerable to overwrites (even worse so).

Best solution so far seems to allocate bootloader on the top of the memory space and reserve bottommost page for single time programmable constants at the time of manufacturing and do a soft check for writepage adress inside the bootloader to protect data stored there against unallowed manipulation.

*Currently working with 18F47J53 and 18F97J94's in some project.

[JPortici]

care to elaborate?
i should check a couple of datasheets but i'm pretty sure that read/write protection fuses can be reset only with the ICSP "erase all" command.
regarding other things... for example setting the oscillator to an external source but no oscillator is actually attached, during startup the oscillator fail bit should be set en automatic switchover to internal oscillator should happen.

[kony]

Except when this automatic switchover is actually not broken (always check your errata  ).

I'd be glad to get prooven otherwise, but so far it does not seem to be the case.

Cited from PIC18F47J53 datasheet (DS39964B):

28.1.1 CONSIDERATIONS FORCONFIGURING THE PIC18F47J53 FAMILY DEVICES
Unlike some previous PIC18 microcontrollers, devices of the PIC18F47J53 family do not use persistent memory registers to store configuration information. The Configuration registers, CONFIG1L through CONFIG4H, are implemented as volatile memory. Immediately after power-up, or after a device Reset, the microcontroller hardware automatically loads the CONFIG1L through CONFIG4L registers with configuration data stored in nonvolatile Flash program memory. The last four words of Flash program memory, known as the Flash Configuration Words (FCW), are used to store the configuration data.
When creating applications for these devices, users should always specifically allocate the location of the FCW for configuration data. This is to make certain that program code is not stored in this address when the code is compiled. The four Most Significant bits (MSb) of the FCW, corresponding to CONFIG1H, CONFIG2H, CONFIG3H and CONFIG4H, should always be programmed to ‘1111’. This makes these FCWs appear to be NOP instructions in the remote event that their locations are ever executed by accident. The four MSbs of the CONFIG1H, CONFIG2H, CONFIG3H and CONFIG4H registers are not implemented, so writing ‘1’s to their corresponding FCW has no effect on device operation.
To prevent inadvertent configuration changes during code execution, the Configuration registers, CONFIG1L through CONFIG4L, are loaded only onceper power-up or Reset cycle. User’s firmware can still change the configuration by using self-reprogramming to modify the contents of the FCW. Modifying the FCW will not change the active contents being used in the CONFIG1L through CONFIG4H registers until after the device is reset.

And yes, this includes program memory readout protection bits.

During flash write, write lock sequence mechanism is used (0x55; 0xAA write to EECON2) - which is the first safeguard against unintenional changes to the (program) memory contents, but inherently has to be disabled due to bootloader trying to achieve exactly this.

IOLOCK and one-way lock for some config bits are valid only for RAM loaded values to serve as a safeguard agaisnt unintentional changes during runtime on the RAM registers stored values.


EDIT:
Thanks for stubble RTFM suggestion.

After re-checking the datasheets of both devices troughly, I managed to find the missing bit (pun intended) in the CONFIGx registers.
It is CONFIG4L -> WPCFG , wich must be cleared along with CONFIG4H -> WPDIS, otherwise all written above stands true.

One should be cautious as this bit invokes protection over whole last page of flash memory, not just the CONFIGx preset values. (This might make for some nasty bug in future, better to prohibit code placement in the last page in linker settings to be sure). So in the end all said previously - except for the soft-checks inside booltoader (when that particular bit is cleared) - holds true.

[JPortici]

jesus
why?
whyyyyy?

I only use K series today, never ever used the J series so i didn't know about that. Again, i may be forgetting about something but i will check tomorrow. it's night time

"workaround" for the oscillator: in my programs the actual oscillator configuration is set between the first 10 instruction. in the bootloader i'm writing for example the very first instruction after the reset vector disable the interrupts. the second one select the FRC as the clock source.