Are there too many different bootloaders in the Arduino world?

[elecdonia]

Problem solved:My USB-to-serial converters were not resetting my pro-mini boards.

It's the series capacitor that gives this result.  While idle DTR =3.6V and \reset =5V.  Then when DTR goes low, the series capacitor forces both both DTR and \reset to drop by 3.6V.  Therefore \reset drops to 1.4V, but this isn’t always low enough to trigger the required MCU reset. If both sides operate at same voltage (either 3.3V or 5V,) then everything works properly. Ideally the adapter should provide power and I/O at the same voltage.

For now all of my pro-mini boards operating at 5V are happy with the 27k resistor added between \reset and ground. Additionally I’ve ordered several USB-TTL adaptors which have 3.3/5V slide switches. I’m hoping that both the Vcc supplied to the target board and the logic levels match up for each position of the selector switch.

[elecdonia]

Could you please post model# and photos of USB-serial converter with switchable output voltages?

Sure.Here's the one I use:
https://www.ebay.com/itm/295528212187
The yellow jumper switches everything between 3.3V and 5V.

Thanks! I am ordering adaptors which look like that. And also another style which has a slide switch for selecting 3.3/5V.

Your “FTDI friend” has jumpers underneath to set supply and I/O voltages.  You should set both the same voltage.  I would pick 5V.

I’ll double check this. I haven’t yet altered its factory-default configuration which is 5V supply and 3.3V I/O. I don’t think it’s I/O voltage is adjustable, but I could be wrong about that. I’ll re-read the adafruit docs and look up the data sheet for the FTDI chip.

So the bootloader wasn't the problem at all.

Correct! This was strictly a hardware issue.

[Peabody]

Your “FTDI friend” has jumpers underneath to set supply and I/O voltages.  You should set both the same voltage.  I would pick 5V.

I’ll double check this. I haven’t yet altered its factory-default configuration which is 5V supply and 3.3V I/O. I don’t think it’s I/O voltage is adjustable, but I could be wrong about that. I’ll re-read the adafruit docs and look up the data sheet for the FTDI chip.

Yep.  See attached pic of the bottom side of the "friend".  Separate options for power and I/O.  Also RTS and DTR (I think either works).

[elecdonia]

Yep.  See attached pic of the bottom side of the "friend".  Separate options for power and I/O.  Also RTS and DTR (I think either works).

We are in sync! 

About an hour ago I took a very close look at my “FTDI friend” USB-TTL adaptor… and discovered that it does indeed have configuration pads on its PC board for selecting either 3.3V or 5V for “I/O logic level.” So I cut the tiny foil jumpering across the “3.3V” pads and put a blob of solder across the “5V” pads. Then I confirmed it is generating 5V logic signals. Next I tested my clone pro-mini boards with it: Now they are all “auto-resetting” properly without needing the extra 22k resistor from \reset to ground.

Previously I had used this same “FTDI friend” adaptor several times without major issues. But this time around it certainly didn’t play well with my new pro-mini boards. Evidently the other pro-mini boards I had 4 years ago worked OK with the out-of-spec \reset signal. Come to think of it I do have a vague recollection that “something” was occasionally flaky about it. But back then I was able to get it to work well enough that I didn’t do any troubleshooting.

FYI, the factory defaults on my FTDI friend were:
   Power voltage to target: 5V
   Logic levels: 3.3V
   RTS connected to \reset. (I’ve never had problems with RTS for \reset so I left this unchanged.)

For all of my current projects the 5V logic levels are totally OK. And now I know how to change it back to 3.3V if I ever need to upload to a target board which cannot handle 5V logic levels.

[Nominal Animal]

This is a common discussion re. Linux distributions ("there are too many Linux distributions") that I have adapted to this subject.
elecdonia, don't be offended, because Q does not refer to you!  This is just a stereotypical adaptation of an analogous subject I've had way too many times.

I only post this because it so well illustrates the problems in posing the kind of question you did, in the subject; and how important it is to ask questions well/precisely.

Q: Are there too many different bootloaders in the Arduino world?

A: No.

Q: But I got some Chinese ones that have really bad bootloaders!

A: That has nothing to do with how many different bootloaders there are.
    You should have asked, "Are some of the Arduino-compatible bootloaders utter crap?", if you wanted a Yes for an answer.

Q: But the world would be so much easier, if there were fewer bootloaders!

A: Each bootloader (except for the crappy ones and bad clones) works well for somebody, so what you are really asking is for others to take a hit so that your own life would be easier.
    It would be a different thing, if you asked for the crappy ones to be burninated with fire; that I would agree with for sure.

Q: No, that's not what I mean.  I mean, it would be better for everyone if there were fewer bootloaders.

A: No.  Each one was written for a purpose, and they work well for that purpose.  Each of them has happy users who do not want to switch.  You think things would be better for everyone, because they would be better for you, so you simply assume they would be better for everyone else too.  After all, it is easier to ask everyone else to change, than change yourself.

    Do you really have the experience to speak for everybody using Arduino?

Q: You're stupid and mean, go away.

A: Aye.

In particular, I myself only use Arduino-compatible microcontrollers with native USB interfaces.  I never use FTDI chips or any other serial-to-USB bridge chips, because I don't need them.  This means that my own bootloader needs are different than those who use microcontrollers without native USB.
I particularly like bootloaders that are exposed using a HID interface (rather than a USB serial interface as is more common), because the latter has all kinds of driver issues (especially on Windows), whereas the former never requires any drivers at all.  The downside with HID interface is the limited bandwidth, so uploading very large firmwares can take a few seconds.
There are some that prefer MTP or even USB Mass Storage interface (so that to upload a new firmware, one just copies the new firmware file to the disk drive the MCU exposes), but I like the HID interface for its simplicity.

[westfw]

Ob xkcd:

[Nominal Animal]

Ob xkcd:

Unvisible for me.  Assuming it is the one I guess it is, there are standards that need to be burninated with fire, and yet I don't think there are too many standards in the world.

[elecdonia]

To shed light on this “too many bootloaders?” question, the AVR device family is large. There are nearly 100 different devices in it. Not every device requires a different bootloader, but there certainly is a huge diversity of features, clock speeds, etc. which correlate to requiring numerous different (device-specific) “fuse” settings.

The “boards.txt” file largely automates the byzantine and confusing complexity of getting all of these items configured properly.

One thing which stands out in every forum discussing AVR devices is all the questions about programming, bootloaders, and proper fuse settings which are posed by new users. As a seasoned professional I’m still at a loss sometimes when I don’t seem to be able to get anything to work right.

Now that my programming hardware is finally working correctly I’ll be testing MiniCore with all the different AVR parts listed in its “boards.txt” file. Having used AVR devices myself since about 2002 I’m pretty sure I have at least one of every part # listed. This includes the really old ones like m8 and m88.

With the mega328 alone there are 3 distinct varieties: mega328, mega328P, and mega328PB. Each possesses considerably different features and internal peripherals.

This is why “boards.txt” has so many entries in it.

[elecdonia]

In particular, I myself only use Arduino-compatible microcontrollers with native USB interfaces.  I never use FTDI chips or any other serial-to-USB bridge chips, because I don't need them.  This means that my own bootloader needs are different than those who use microcontrollers without native USB.
I particularly like bootloaders that are exposed using a HID interface (rather than a USB serial interface as is more common), because the latter has all kinds of driver issues (especially on Windows), whereas the former never requires any drivers at all.

+100 on this.

I’ll take a close look at my bin of Arduino style boards. I probably have at least one board which is “HID.”

[westfw]

With the mega328 alone there are 3 distinct varieties: mega328, mega328P, and mega328PB. Each possesses considerably different features and internal peripherals.

Atmega328 and ATmega328P are essentially identical.  The only programming difference that anyone has found is that the P version permits the brown-out detection to be disabled by software (for lower sleep-mode power consumption.) (and a rather inexplicably different device signature.)  Atmel was ... not terribly logical with their naming standards (the PB should surely have been called something new.)
Microchip's new (?) AVRxxxDynnn looks more promising, actually.

Until miniCore came along, "recommended practice" for users who accidentally bought the non-P 328 chips for their Arduino projects was to burn the default ATmeag328p Optiboot. Since optiboot hardwires the device signature, thereafter the chip would identify itself (when using the bootloader) as a 328P, keeping Arduino happy without having to have explicit support for a different chip.

[westfw]

BTW, there's a new serial protocol aimed specifically at bootloading (the stk500 protocols were all intended to talk to hardware programmers), designed by Stefan Rueger (the author of avrdude.) This enables a bootloader even smaller than Optiboot (it no longer has to keep creating "fake" responses to some commands), or allows even more features to fit in 512 bytes.

I've thought something like this should exist for a long time, but it's one of those things where the people writing the host-side software (avrdude, in this case) had to buy into the idea first.

https://github.com/stefanrueger/urboot

[elecdonia]

The “P” in mega328P might be there because of the Atmel marketing department. The initial “P” stands for “pico power.” Even though earlier ancestors in the AVR family also had impressive “low-power” capabilities, Atmel really began marketing “ultra-low-power” with the 328P.

Then the 328PB came along. It is significantly different, beefed up with additional internal hardware, such as a second USART port. Some folks are unhappy that the 328PB no longer contains the “full swing crystal oscillator” option. They prefer the “full swing crystal oscillator” for applications in electrically noisy environments.

[elecdonia]

BTW, there's a new serial protocol aimed specifically at bootloading:
https://github.com/stefanrueger/urboot

Thanks! I took a quick look at GitHub. I like the concept and I will try it soon.
     Does it require adding a “programmer” entry (named urclock) to the Arduino IDE?

[westfw]

Does it require adding a “programmer” entry (named urclock) to the Arduino IDE?

The "programmer" field are used only for "upload using programmer" or "burn bootloader."  I suppose that that might be one way of trying out a new bootloader protocol (via the "upload using programmer.")  But normally I think you'd use new boardname.upload.* parameters in the boards.txt file.

[elecdonia]

Does it require adding a “programmer” entry (named urclock) to the Arduino IDE?

The "programmer" field are used only for "upload using programmer" or "burn bootloader."

I understand. The "programmer" for burning "urboot" into AVR chips will continue to be my "Arduino-as-ISP" device. I've used this method for years to upgrade bootloaders on Arduino-style boards and to program raw AVR devices. I've also used a ***real*** STK500 board, but the 2 of them I own are getting long-in-the-tooth (when did I buy those? 1998? 2001? Gosh, I even have an STK200. I have no idea when or how I ended up with that one!).

I recently picked up a nifty $4 USD Chinese "OPEN-SMART" shield. I parked it on top of a (clone) Arduino UNO, which runs the (slightly modified) "Arduino as ISP" sketch provided by the folks who created the OPEN-SMART board. The OPEN-SMART board has a buzzer and this modified sketch makes it beep at the end of a successful programming event.  I also added a 10uF capacitor from the UNO's RST pin to ground in order to protect the host UNO from auto-resetting. This prevents accidental uploads into the host UNO underneath the OPEN-SMART shield.

A cool feature of OPEN SMART is its ZIF socket for programming 28-pin DIP package AVR chips. I have plenty of 28-pin mega328P chips which I'm using on "Transistor Tester" boards. The "Transistor Tester" firmware doesn't use a bootloader, so I program those devices directly with AVRDUDE. I'm also setting up several other mega328P chips with MiniCore/Optiboot for other projects where the MCU is on my own board and I will be using the Arduino IDE for development/debugging. I should mention that I began writing code for AVR chips before the Arduino project existed. That was  with Atmel Studio and...  (groan ) ..writing code in assembly language. The target device was often an ATtiny13. It's hard to believe I did that 20 years ago...

At this time I'm still in the process of  setting up several Arduino-style boards with MiniCore/Optiboot and also putting it on some "bare" mega328P chips. At a later date I'll experiment with "urboot." 

[elecdonia]

Then your project will not have an Arduino bootloader and you will have to re-do all your Arduino work.

This is completely false.  The Arduino bootloaders are all completely separate and non-interacting with the user applications, whether they are "Arduino sketches" or programs written in some other development environment.  You can write Arduino sketches without using the bootloader (saves 0.5 to 8k of memory) (there's even an "upload using programmer" command), or you can use the Arduino bootloaders with non-Arduino applications.

    AVR chip with bootloader:

1) Good:   If the target board has a serial port then it can easily receive new firmware straight from the Arduino IDE or from a PC running AVRDUDE. Many of my applications already have a serial port for external communication. Therefore adding bootloader capability is "zero overhead" on the hardware side 
2) Bad:     Each time the target board powers up or resets there is a delay (~1 sec) before the application starts up. The bootloader always starts first, but then times out and runs the application after determining that it isn't connected to AVRDUDE
3) Could be an issue?  The bootloader reduces the amount of flash memory available to the application by 512 bytes

     AVR chip without bootloader:

1) Good: Application on target board starts up immediately after power-up or reset
2) Also good: All of the flash is available to the application
3) Also good: Arduino IDE can still be used for development/debugging via the "upload sketch with programmer" button in the Arduino IDE
4) Bad:   Firmware updates require a 6-pin ICSP header on the target board along with a cable connected to a programming device. This may be acceptable for applications which never (or rarely) require firmware updates.

[westfw]

2) Bad:     Each time the target board powers up or resets there is a delay (~1 sec) before the application starts up.

One of the modifications made to the bootloader very early in the development timeframe was the "Adafruit fast start" mod.  This detects the RESET cause, and only runs the bootloader for "external RESET signal", but NOT for power-on reset.  So there shouldn't be any start-up delay except when you actually use the reset button or Arduino auto-reset feature (any connection/change of parameters on the serial port.  Sigh.)

OTOH, I've seen AVRs be bit confused about their reset cause when powered on, depending on maybe PWR rise time, exactly what is connected to the RESET pin, and similar.  So if you're dependent on the fast startup, eliminating the bootloader is a good course of action.

[elecdonia]

2) Bad:     Each time the target board powers up or resets there is a delay (~1 sec) before the application starts up.

One of the modifications made to the bootloader very early in the development timeframe was the "Adafruit fast start" mod.  This detects the RESET cause, and only runs the bootloader for "external RESET signal", but NOT for power-on reset.  So there shouldn't be any start-up delay except when you actually use the reset button or Arduino auto-reset feature (any connection/change of parameters on the serial port.  Sigh.) OTOH, I've seen AVRs be bit confused about their reset cause when powered on, depending on maybe PWR rise time, exactly what is connected to the RESET pin, and similar.  So if you're dependent on the fast startup, eliminating the bootloader is a good course of action.

I have observed faster start-ups of my application code at power-up. Now I'll have to actually measure that...

That said, because a majority of my applications have a serial port connected to something else (such as a PC), I still must confront the issue of "what happens after the serial port hardware spits out an auto-reset pulse?"

But... (I'm a hardware guy) How about putting a simple spst switch somewhere on the target board which enables/disables auto-reset signal from reaching the MCU? Before uploading firmware the switch must be set the "bootloader enabled" position.

I'm already using this (in a way) on my "Arduino UNO as ISP" device. The 10uF cap I attached from the UNO reset pin to ground prevents the serial port auto-reset pulse from being detected by the MCU on the UNO board. I disconnect the 10uF cap if/when I need to upload a modified "Arduino as ISP" sketch to the UNO. Actually the capacitor can be as small as 1uF. It only needs to be 10x larger than the 100nF auto-reset capacitor to prevent the negative-going auto-reset pulse from getting detected as \reset by the MCU on the UNO board.   

[Doctorandus_P]

Re: Are there too many different bootloaders in the Arduino world?

Yes, there are far to many. Even a single one is one too many.
I never bothered using any of them, I just plug in a programmer and it always works. Regardless of modes, no silly states to enter boot loader mode, no waiting, no hangups. You just command the programmer to reset the uC and reprogram it and it does it. Always.

Bootloaders do have some limited use, for example for firmware updates in the field where a programmer is not always available, or when a mars rover on another planet has to receive a bug fix, but I quite despise the things for general development on your own desk.

[westfw]

How about putting a simple spst switch somewhere on the target board which enables/disables auto-reset signal from reaching the MCU?

Sure.  The oldest Arduinos didn't have the auto-reset feature at all, and users had to manually hit the reset button before they did an upload.  That was when the timeout was closer to 10s, though.  If you want to operate that way, you should probably rebuild the bootloader for a longer-than-1s timeout.


The "megaTinyCore" (tiny-0/1/2) bootloader added a feature that runs the other way - ONLY run the bootloader at poweron.  That's because the limited pin count would otherwise require you to give up a GPIO pin for the RESET function.

I just plug in a programmer and it always works.

Always?  Never tried to ISP program a chip that was in DEBUGWIRE mode?  Never had conflicting hardware on the ISP pins?  Never found out that the new chip you were planning on using needs HVSP or TPI or PDI that your programmer doesn't support?  Never mis-programmed your  fuses and now you need HVPP or an external clock to recover?

[Siwastaja]

Besides, bootloaders are mystified for no reason. If one is capable of writing a non-trivial (more than led blinker) firmware, they surely are capable of writing their own bootloader. The reason to do is exactly the same why we write our own software and firmware at all - to get exactly the functionality we need.

This is also why so many exist, and why some are so crappy - they are easy throwaway projects. If you feel like you need a bootloader and can't easily find something suitable, maybe just write your own.

[elecdonia]

The oldest Arduinos didn't have the auto-reset feature at all, and users had to manually hit the reset button before they did an upload.  That was when the timeout was closer to 10s, though.  If you want to operate that way, you should probably rebuild the bootloader for a longer-than-1s timeout.

Yes, I'm familiar with adjusting the length of the bootloader timeout. That said, I want my finished projects to start up as fast as possible after reset or power-up. I think I will place a 2-pin header on my boards where "enable bootloader" requires shorting these pins with a jumper. I don't expect end-users to upload new firmware very often, but I do wish to make this possible. 

The "megaTinyCore" (tiny-0/1/2) bootloader added a feature that runs the other way - ONLY run the bootloader at poweron.  That's because the limited pin count would otherwise require you to give up a GPIO pin for the RESET function.

This is good to know. I haven't recently used any Tiny AVR but I fully agree that conserving every possible GPIO pin is a priority. In the past I mainly programmed Tiny devices with ICSP. I don't foresee many Tiny AVR projects where I would wish to have a bootloader on them. One of my designs with ATtiny13 is still in mass-production by my former employer with 12-year old firmware which still functions perfectly. It does require every available GPIO pin, so reset was always disabled. It does not contain a bootloader.

On a related topic I've been bashing my "OPEN-SMART" (clone) ISP-programmer shield intensively. I did some mods to the shield which I will fully document after more testing.

My first mod was to prevent the UNO board (underneath the OPEN-SMART shield) from processing reset commands received from its USB serial port. This was done by placing a 10uF capacitor from UNO board reset to ground. This isn't strictly necessary because the standard "Arduino-as-ISP" driver sketch operates at 19.2k baud vs. the 115k baud UNO board bootloader. But when I was "playing around" with serial port speeds (running everything at 115k) I accidently uploaded a sketch intended for the target AVR into the UNO board. Because I dedicated this particular UNO board to permanent "Arduino-as-ISP" duty I decided to disable its bootloader via hardware.

My second mod relates to using the 6-pin header on the OPEN-SMART shield to connect a USB-TTL adaptor such as "FTDI-friend." This permits uploading sketches into a target AVR installed in the ZIF socket (naturally this works only after installing the bootloader into the target AVR device). It's not likely to get used by most people. But the OPEN-SMART folks built their shield with this feature so I decided to try using it. One important part was missing: The 10k pullup resistor from the target AVR reset pin to +5V. The OPEN-SMART shield comes with the standard 100nF capacitor from DTR to reset. But without the 10k pullup resistor it barely works. With the 10k resistor it works great.

Mod number 3 is adding an LED and series resistor to the "LED-pin" (aka D13) of the target AVR in the ZIF socket. Other small Arduino boards (pro-mini) have this LED. I like having it because it "double-blinks" after reset/power-up, confirming the bootloader started up. It also permits running the standard Arduino "blink" sketch.       

[elecdonia]

With my OPEN-SMART AVR programming shield fully functioning I’m checking out all the 28-pin DIP AVR chips in my hoard of MCU devices.  I have ATmega 328P, mega8, mega88, mega48, and maybe some other ones. With jumper wires running to a 40-pin socket on a breadboard I’m also testing 40-PIN DIP parts like the mega644 and mega1281.

     People might ask “Why are you tinkering with all these old DIP-package parts?”

The answer is that I’m primarily a hardware guy. I’m very comfortable working with “through-hole” construction, especially for one-off hand-built projects, having made things that way since the 1970’s. Also I have plenty of these older AVR parts on hand because I’ve been using them since the year 2001 or so.

My routine is to put Optiboot on these bare AVR chips.
I’m setting their fuses for 8MHz internal clock oscillator. For me this always seems to work fine with a 38.4k serial port rate, which is fast enough for the projects I’m developing at the moment.
I also adjust fuses for “EEPROM data preserved” and “application can read (only) from boot area.”
(I edited a few lines in the appropriate boards.txt files to automate this for the AVR part numbers I’m using.)

After installing optiboot then I access the target devices through the serial port via the old-style Arduino IDE (v1.8.9 I think), and for “board” I’m using the appropriate core from MCUdude.

Next I run this “FuseBytes” sketch to double-check that all the fuses are exactly the way I want them to be:
   https://github.com/WestfW/fusebytes
(Note who the the author is: Thanks!)

[westfw]

Next I run this “FuseBytes” sketch to double-check that all the fuses are exactly the way I want

I don't think "FuseBytes" handles all the different chips you are using.  But it's been a long time since I looked at it.

[elecdonia]

Next I run this “FuseBytes” sketch to double-check that all the fuses are exactly the way I want

I don't think "FuseBytes" handles all the different chips you are using.  But it's been a long time since I looked at it.

I’m adding them as I go. While doing this I’m gaining an understanding of the structure of the fusebytes sketch and the internal structure of the AVR device itself. There aren’t a lot of other “example” Arduino sketches out there which show how to read from “special” AVR data storage locations such as the signature, fuses, and lock settings.

Also most of my projects use text-based serial communication, preferably in human-readable formats. So I’m always looking for apps like fusebytes to study and adapt the concepts to the code I’m writing.

Yet another reason I’m doing this is to refresh my understanding of the AVR device fuse parameters themselves and how/when to use different settings. Getting them right can challenge even an experienced hardware/software engineer. I can see why so many newbies hit a brick wall, especially when trying to get bare factory-fresh AVR devices set up and working with the bootloader for the first time.