Best way to update microcontroller firmware for end user?

[bson]

Too bad WebUSB never saw universal adoption - it was perfect for firmware updates, configuration, grabbing device state for bug reports, etc.

[Scrts]

Bootloader that has USB as mass storage device. Connect to PC -> gets up as flash drive, you drop the hex or bin and it automatically flashes if it's later version than current.

[WattsThat]

OP, your first sentence said “open source” project. Do you expect users to be making custom, one off versions? If not, then the only way you can be sure of what gets loaded into your device is for you to provide the hex file. Supporting end users re-compiling source in an unknown version environment, especially Arduino, is stuff nightmares are made of but it can be somewhat avoided if you’re careful by providing the required versions and the actual library sources.

I don’t like the SD card idea, it too requires a level of user ability that you cannot be sure of unless you mail imaged cards to users and I don’t think you’re planning on doing that, right? Then you have the additional bom cost and software overhead. How much flash space do you have available? Another day, another K consumed with used once a year overhead...

I don’t think using usb is any great hardship for most users, is it really any more difficult than requiring the SD card and writer, downloading a specific filename to the card then putting card into your device and somehow get it to load?

All things considered, a command line invocation of avrdude is not that difficult. The more effort you put into to making it idiot proof only creates more idiots that you have to support. Set the users expectations when you sell it and be done with it, unless of course it’s selling for hundreds or thousands of dollars. If that’s the case, then I’d be looking at this from a very different perspective 

[mariush]

I agree that SD cards seem like a bad idea.
You have to put a SD card holder on the board which uses space and costs money,
you have to add the functions and all that's needed in the bootloader to support it,
you probably will have to deal with the various formats of SD cards and you'd probably have to ask people to buy 1-2 GB cards which are harder to find these days
you'd have to ask people to format them in FAT16 or FAT32 so that you can read them (you're losing people at this point)
You're assuming majority of people actually have SD card readers ... not all people have laptops with sd card readers built in, or separate card readers... how many would buy a SD card and sd card reader just to update the firmware

Going to my previous suggestion of adding a 32-64 KB eeprom chip on the board which uses very little space and costs very little (under 20-30 cents in volume) and only ties up a couple i2c or spi pins (not really as you have chip enable pins so those data/clock pins could be shared with other stuff.
You could have a jumper or button which when set/held down when device starts puts your device into programming mode.

Basically make your device show up as a serial port and respond to just 3-4 commands

v - version : reply with a version string so your firmware updater can detect your device listens on that com port  (ex "my device name version=0.1.100 date=20200329")
Your firmware updater tool can extract version 0.1.100 and date from reply if needed.

h / help / ?  : show a help message ex. "my device name firmware update interface. Remove jumper or release button at power on to resume normal operation.  See documentation to update your firmware"
 
w : write page to eeprom followed by 1 byte which represents the page number (256 byte pages) and optionally number of bytes (you can always pad with nulls to always send 256 bytes at a time)
after 256 bytes you can send a 2-4 byte crc checksum and your bootloader can reply with ok or fail if the checksum doesn't match

r: read page from eeprom (so that your firmware updater can double check that the page was written correctly)

At next reboot, your bootloader can check the first few bytes in the eeprom to compare the version of the new firmware and also reads the firmware size from there and can do checksums if desired and then update the internal flash memory in one shot

You only have to set the USB as com port, or have a separate serial port ... if they're programmers they will have a usb to serial or they can buy a usb - serial ttl cable / adapter

[kripton2035]

Quote from: Mr. Scram on Yesterday at 22:57:25

>Quote from: kripton2035 on Yesterday at 18:50:20

using Xojo for 10+ years I can say yes I know how to build and maintain win-mac-linux apps.

Can you tell us more about how you do that?

all you need is here : https://www.xojo.com/
you can play with it all you want, you need a paid licence only to build a standalone app.

I was hoping for some practical experience and what works well and what doesn't, instead of just the manufacturer sales pitch or myself bumbling about. 

well there are some functions that are specific to each platform. usb is one of them, so you have to program each platform differently, but the IDE integrates it very nicely.
if you use tcpip for linking your devices, it is the exact same code for all the platforms.
there are examples in the app that shows you the same app in different platforms, even including a webapp and a mobile (ios) app.
the android version is hoppefully coming this year(?)

[NorthGuy]

I have some trouble seeing how it wouldn't complicate matters significantly. You have to have three development tool chains or find one which does all three and does it correctly. You have to figure out how to test your product on all platforms, which includes a multitude of subvariants and endless software and hardware variations. Just the Windows platform is fairly diverse and Linux takes this to another level. After release people will invariably find all kinds of ways to break your product and exceptions and you're expected to support and fix that too.

You exaggerate too much. Of course you need to test a lot, and nonetheless, once you distribute to thousands of people you they will find (and hopefully report) bugs. But this is not such a big deal.

Microsoft is very good at backward compatibility. I used to sell a relatively big software program. The development was abandoned long time ago and the last supported platform was Windows XP. It hasn't been touched since. But it still works on all the newest Windows without problems.

Linux is good too. I can still run programs which I wrote and compiled in last millenium. This went through different Kernel versions, different distributions - from all Red Hat and my minimalist Linuxes to the most recent Debian. I never had to recompile.

Apple are cowboys. Breaking older software means nothing to them. So, you're likely to have problems when a new version appears, but you still can deal with that.

The modern over-bloated software with uncontrolled dependencies has a tendency to be buggy and vulnerable to configuration changes. Modern software development principles encourage such approach. But it's all up to you. It is nothing wrong if you think about testing and support before you develop your software.

At any rate, it's much better to have bugs in your PC software, which you can update in a blink of an eye, rather than in your bootloader, which you absolutely cannot fix. Therefore, an approach with a lean bootloater and most of the processing shifted to PC is much more robust.

[ogden]

You exaggerate too much.

Exactly. Open source product is aimed at tinkerers, thus providing just (nonbloated) Windows app is fine, IMHO. MAC and Linux users usually know how to run windows apps, right?

[nctnico]

I have some trouble seeing how it wouldn't complicate matters significantly. You have to have three development tool chains or find one which does all three and does it correctly. You have to figure out how to test your product on all platforms, which includes a multitude of subvariants and endless software and hardware variations.

It's probably less of a hassle if you already have the tool chains already set up, but it seems less than trivial if you don't already have a fair bit of it set up.

The QT and wxWidgets frameworks (combined with GCC) make this process pretty much painless because they hide most of the idiosyncrasies of the underlying OS (which is a good reason to use these frameworks even if you don't plan to make cross-platform software). Other than that supporting an OS does takes testing (easely done using VMs nowadays) and has different packaging setups. I have some projects where the embedded Linux version (running on ARM) compiles from the exact same sourcecode and IDE as the Windows version. But there is one thing I always do: link all the libraries statically together with the executable and if that isn't possible supply the libraries with the executable. Relying on libraries being available on the system is asking for problems.

[Mr. Scram]

You exaggerate too much. Of course you need to test a lot, and nonetheless, once you distribute to thousands of people you they will find (and hopefully report) bugs. But this is not such a big deal.

Microsoft is very good at backward compatibility. I used to sell a relatively big software program. The development was abandoned long time ago and the last supported platform was Windows XP. It hasn't been touched since. But it still works on all the newest Windows without problems.

Linux is good too. I can still run programs which I wrote and compiled in last millenium. This went through different Kernel versions, different distributions - from all Red Hat and my minimalist Linuxes to the most recent Debian. I never had to recompile.

Apple are cowboys. Breaking older software means nothing to them. So, you're likely to have problems when a new version appears, but you still can deal with that.

The modern over-bloated software with uncontrolled dependencies has a tendency to be buggy and vulnerable to configuration changes. Modern software development principles encourage such approach. But it's all up to you. It is nothing wrong if you think about testing and support before you develop your software.

At any rate, it's much better to have bugs in your PC software, which you can update in a blink of an eye, rather than in your bootloader, which you absolutely cannot fix. Therefore, an approach with a lean bootloater and most of the processing shifted to PC is much more robust.

Maybe it helps to explain my reasons. Though my work has been in somewhat different areas I've consistently seen a lack of standardization or a pluriformity of standards become support hell. Picking a well chosen standard and then sticking to it is invaluable in my experience. Failure to do so invariably bites you years and even decades later. Taking shortcuts is often rewarded with paying the gains many times over. That's why diving into a development cycle for three major platforms to create a product which isn't the core business gives me the heebie jeebies. You're committing a lot of time and resources to something which may prove to be a major and ongoing distraction. That can be annoying and costly for your average outfit. It can be much more disruptive if you're a one man show.

[Mr. Scram]

The QT and wxWidgets frameworks (combined with GCC) make this process pretty much painless because they hide most of the idiosyncrasies of the underlying OS (which is a good reason to use these frameworks even if you don't plan to make cross-platform software). Other than that supporting an OS does takes testing (easely done using VMs nowadays) and has different packaging setups. I have some projects where the embedded Linux version (running on ARM) compiles from the exact same sourcecode and IDE as the Windows version. But there is one thing I always do: link all the libraries statically together with the executable and if that isn't possible supply the libraries with the executable. Relying on libraries being available on the system is asking for problems.

That's what I see many parties doing and although it leads to a lot of bloat it seems to prevent many of the library headaches. Dependency hell remains an issue.

[kripton2035]

to compare with Xojo, you check the box you need : Windows, Mac and/or Linux, choose 32 or 64 bits, clic build. and Voilà.
the libraries are build in with the app on mac, and in a folder beside the app for windows and linux.

[NorthGuy]

Though my work has been in somewhat different areas I've consistently seen a lack of standardization or a pluriformity of standards become support hell. Picking a well chosen standard and then sticking to it is invaluable in my experience.

That's exactly what you do. You standardize your UI. Then you use OS-specific parts (through third-part libraries or otherwise) which provide a standard interface to your application. Your application uses this common OS-indepent standard interface, so your source code is practically independent of the OS. If some OS-specific peculiarities come into way, you just fix the corresponding OS-specific part, and voila - all your applications are fixed.

[Mr. Scram]

That's exactly what you do. You standardize your UI. Then you use OS-specific parts (through third-part libraries or otherwise) which provide a standard interface to your application. Your application uses this common OS-indepent standard interface, so your source code is practically independent of the OS. If some OS-specific peculiarities come into way, you just fix the corresponding OS-specific part, and voila - all your applications are fixed.

That OS specific part seems to be a rather large part. Just the testing alone is a fair bit of work, even if you could do it all in VMs. Even the big boys seem to shy away from supporting too many platforms. I'll have a tinker with this sometimes soon but compiling is obviously not the whole story.

[nctnico]

That's exactly what you do. You standardize your UI. Then you use OS-specific parts (through third-part libraries or otherwise) which provide a standard interface to your application. Your application uses this common OS-indepent standard interface, so your source code is practically independent of the OS. If some OS-specific peculiarities come into way, you just fix the corresponding OS-specific part, and voila - all your applications are fixed.

That OS specific part seems to be a rather large part. Just the testing alone is a fair bit of work, even if you could do it all in VMs.

Testing is not a lot of work. The primary function of the software is the same so that doesn't need to be re-tested. Platform specific testing is limited to checking the installation, paths and whether the GUI is rendered correctly. After all the platform independent framework already takes care of the platform specific stuff. wxWidgets for example has an API which uses configuration files on Linux and the registry on Windows without needing any change on the application's side.

[Mr. Scram]

Testing is not a lot of work. The primary function of the software is the same so that doesn't need to be re-tested. Platform specific testing is limited to checking the installation, paths and whether the GUI is rendered correctly. After all the platform independent framework already takes care of the platform specific stuff. wxWidgets for example has an API which uses configuration files on Linux and the registry on Windows without needing any change on the application's side.

I guess it depends a bit on how thoroughly you want to test each release. Assuming too much there is a bit of a risk. The initial setup seems significant regardless. You need to set up the tool chain, VMs and testing. Once you're settled things seem to cost a somewhat less significant amount of effort. Again, that's for a product you're not even selling. I'm not trying to be a contrarian here but if things really were so effortless I'd expect almost anyone to support every imaginable platform. I'm curious how OP currently feels about the various options.

[NorthGuy]

That OS specific part seems to be a rather large part.

It depends on how much you want to bring from the OS.

[nctnico]

Testing is not a lot of work. The primary function of the software is the same so that doesn't need to be re-tested. Platform specific testing is limited to checking the installation, paths and whether the GUI is rendered correctly. After all the platform independent framework already takes care of the platform specific stuff. wxWidgets for example has an API which uses configuration files on Linux and the registry on Windows without needing any change on the application's side.

I guess it depends a bit on how thoroughly you want to test each release. Assuming too much there is a bit of a risk. The initial setup seems significant regardless. You need to set up the tool chain, VMs and testing. Once you're settled things seem to cost a somewhat less significant amount of effort. Again, that's for a product you're not even selling.

You have to set it up once for the first product. For the nexts products it is rinse & repeat. Bonus points for compiling several different products from the same source.

[westfw]

Come on.  The avr side bootloader is somewhere between 512 bytes and 4K of code.  A host side app to send it a new program shouldn’t be that hard.  Use python, or one of the other explicitly cross-platform languages, and skip the GUI.  Yeah, there may be corner cases that won’t work, and they can fall back to using the arduino IDE.

[donotdespisethesnake]

I've got a open source project I'm busy with which uses a AtMega32u4. I plan on producing and selling a small number of PCB's with this microcontroller which will have a 6 pins  installed for installing the bootloader via SPI. Once that's done I can upload the latest code via usb/arduino ide and then ship out.

Simple answer : Arduino IDE. It's already a cross-platform programmer with loads of support on the net. I would expect open source users who want to tinker should be able to cope with Arduino IDE, it's designed for non-technical people after all. There will be some support issues but is considerably less effort than developing your own cross-platform tool and supporting that. Typical user errors are with cabling anyway.

If you had a commercial product where the users are completely non-technical, then micro SD is a good option. You need to squeeze in FatFs, which might be a challenge with mega32. I wouldn't recommend a raw image, users will then need a tool to write images.

There is no point doing extra stuff with plug in programmers and additional EEPROMs. Instead of that just use a bigger chip with the capability onboard.

Also, it's worth considering other chips apart from AVR, although that may be something for future projects, since you will probably end up redoing a lot of stuff. STM32 could be a good option as they are somewhat supported in the Arduino ecosystem, otherwise Mbed has an easy firmware upload method.

If you happened to use LPC1769, there is already a open source SD bootloader. https://github.com/triffid/LPC17xx-DFU-Bootloader

[Peabody]

ST offers offers a small program called Flash Loader Demonstrator that flashes a hex file to an STM32.  That would be a lot easier than dealing with the Arduino IDE.  But I think it's only for Windows.

But for the Atmega chips, it seems Avrdude would be the way to go. You would distribute a ZIP file containing Avrdude, the hex file and a batch file containing the freakish Avrdude command line, and hopefully the user would just connect via USB and run the batch file.  I suggest this never having tried it myself, but it seems like it ought to work.  Installing and dealing with the Arduino IDE is asking a lot of someone who doesn't need it otherwise.

[MarkL]

ST offers offers a small program called Flash Loader Demonstrator that flashes a hex file to an STM32.  That would be a lot easier than dealing with the Arduino IDE.  But I think it's only for Windows.
...

That's another implementation of DFU.  That's what I have my clients use.  DFU has already been mentioned a couple of times in this thread and is a standard, multi-vendor way to load firmware over USB.  It's been around for many years.

FLIP is Microchip's implementation of DFU with a GUI.  And Microchip also provides pre-compiled DFU bootloader binaries specifically for the OPs AtMega32u4 target.

And if FLIP isn't suitable for some reason, avrdude also implements DFU.  Or you could even use the raw dfu-util or dfutool. 

I'm just not seeing the necessity of adding an SD card or getting into multi-platform development solely for the purpose of updating firmware.

EDIT: "Flash Loader Demonstrator" is not DFU, my apologies.  STM's DFU loader is called "DfuSe Demo" (aka. STSW-STM32080).   Everything else is still correct.  I had the incorrect package name.

[SiliconWizard]

You've been given a few options here in this thread.

The one you should choose entirely depends on what your project exactly is IMO.
You said "I plan on producing and selling a small number of PCB's with this microcontroller", implying that you're basically worrying for what would amount to a small number of products - which are not even full products, and let others use their own tools. I really don't think you should bother. If it's Arduino-based and it's an open source project, just let people handle this with the Arduino tools.

Another point to consider, of course, would be how often you'd reasonably envision your firmware to be updated. If said firmware is stable when you release the boards, there isn't necessarily a reason for the users to update. It's like we've been completely brainwashed by all this software updating frenzy. If it works as advertised when sold, you don't necessarily have to allow your users to update. Just a side thought.

Anyway, I would only bother easing the updating process if I was planning on selling fully-assembled products (which you could call "end-products"), not in your case. Just MHO though.

[Warhawk]

tl;dr;
Please consider releasing a product, that is properly tested during the development and works out of the box. Many of us are already sick of constant firmware updates and half-baked products. For instance, I have a Siemens me45 next to me and it has never needed any firmware update. Firmware update is the last resort and it should not be a part of the product lifecycle. Please accept the responsibility of an engineer and do it right.
 

[Siwastaja]

Case A)

It's a fairly simple device, which only connects to the outside via its own physical interface (a screen, buttons), i.e., no digital communication bus nor a removable media storage in normal operation:

1) Do it properly so that no updating is necessary,
2) Updating requires a special measure, and it's gonna suck. Just minimize the cost. For example, instead of needing to sell the user an Arduino, sell them a $1 USB-TTL cable widely available from Ebay; write a super simple UART bootloader, and a custom flasher tool for a few most popular platforms like Windows, Mac and linux. If the MCU has built-in USB, then using USB is a no-brainer. Your flash upgrader software on the PC side is then the key.

Case B)

It's a device which connects to the world using any kind of digital communication bus: write a bootloader so you can update through that bus, by using a simple update tool program you write for the purpose. Similarly, if it has a removable storage medium anyway, then use it for the flash update.


As you can see, both suggestions involve writing a bootloader. It isn't that hard, typically around 100-200 lines of code on the MCU side, and maybe 200-300 on the PC side when you do your custom firmware update software. Learn the idea once, and you have unlimited possibilities in the future. It's a great value-add.

[Nominal Animal]

ATmega32u4 is one of my favourite microcontrollers, and has a native low-speed/full-speed (1 or 12 Mbit/s, respectively) hardware USB interface.

All your users need for flashing a new firmware, is a normal USB application.

The Arduino environment uses the AVR109 protocol for uploading firmware to '32u4-based microcontrollers.  (The AVR910 protocol with hardware SPI pins and another microcontroller or USB-SPI bridge is used when you wish to reprogram the microcontroller completely, including the bootloader.)
You do not need the entire Arduino environment, as it uses avrdude for flashing.  You can just implement the protocol yourself.  (Meaning, your own standalone single-file firmware updater would be a normal application, that flashes the Arduino .hex file to your microcontroller.)

The protocol uses USB serial, and is easy to implement.  The "trick" is that depending on the bootloader, the '32u4 will only respond to the protocol for a short while after a reset, or if a specific pin is in a specific state after a boot.  If your device implements USB Serial, you can use a baud rate change to switch to firmware update mode (reset, followed by USB enumeration, exporting an USB Serial device used for the AVR109 protocol, like Arduino Leonardo boot loaders); if your device implements USB HID, you can use a custom HID command (from host to device) for the same purpose.  If you write your own bootloader, like PJRC did (proprietary HalfKay, for Teensy 2.0 using ATmega32u4), you can even use HID for the bootloading (albeit a bit slower than USB Serial, just a few kbytes per second).

The bootloader used on Arduino Leonardo and Pro Micro clones is Caterina, which is based on Dean Cameras LUFA library.  You might to take a look at LUFA; there might already be tools you can use there.

On my Linux machine, I use a wrapper around avrdude when flashing my Pro Micro clones (ATmega32u4, same pinout as SparkFun Pro Micro, uses Arduino Leonardo bootloader) which uses the baud rate change (to 1200 baud) to auto-reset the clone, and waits until the USB Serial endpoint gets enumerated, then runs proper avrdude with the corrected parameters (reflecting the re-enumerated USB Serial end point).  That way I only need to reset the clone (shorting RESET pin to GND) if my firmware does not use USB Serial (as then there is no USB Serial device to change baud rate for).

The thing that is a bit annoying here, is that you really need to write and test the application on all OSes you intend to support.

For Linux, you'll need to compile a version for each hardware architecture (x86, x86-64/AMD64, some ARM variants, perhaps 32-bit MIPS), and link it statically so the same binary will work across all distributions.

Of course, if you want your device to be used with different OSes, you do need to test it on them also, so having the necessary hardware and perhaps virtual machines to test different versions of the OSes, is a good idea.