How does one select an RTOS ?

[tggzzz]

Most often the best RTOS choice is none.

You can do plenty of simple multitasking in a more efficient way using plain C with hot loops, state machines, interrupts..etc.

Sounds good for a simple hobby projects or as a learning experience. Or if you have very limited resources and trying to save a few kb of memory.

In practice, implementing your own multitasking tends to become a chunk of spaghetti with subtle bugs and hacks, especially with multiple developers.

Especially when you factor in the effects of caches (and interrupts) on multi-threaded C code, written by typical developers, who apparently live in Lake Woebegon

[Siwastaja]

Not directly related to my question but is it common to make such a big change (change the architecture) during the lifetime of a normal product ?

Yes and no. You don't switch MCU families like you change underwear, but having the option to redesign into different MCU while having an idea that "oh, it will take two weeks to port" instead of "I have no idea whatsoever, maybe month, maybe a year!?" is a great safety net.

If you write your whole app to depend on features of an RTOS which is not available on different MCU family you have painted yourself into corner. Similarly, if you base your code into availability of vectored, priorized hardware interrupt controller with software interrupts available, like I usually do, this rules out some weird architectures without major modifications, too.

Most important, by far, is the separation of your application logic and peripheral access, using good interfaces which make sense, so that rewriting the peripheral part is trivial. This is totally orthogonal to the RTOS issue. And BTW, this is exactly where stuff like "STM32 HAL" drives you in the wrong direction: now your whole codebase depends on ST's interfaces and you can't switch, not even to different STM32 families without doing modifications here and there as the underlying hardware leaks to the abstractions. What you  can do, instead, is to write abstractions that match your application exactly, and then the underlying peripheral code is easy to rewrite.

[metebalci]

I had to choose an RTOS, and I ended up with NuttX.

I wanted/needed a complete package with not only the kernel but also stacks and drivers for Ethernet and TCP/IP and USB, and I absolutely didn't want to mix and match products from different vendors for support reasons. This eliminated FreeRTOS, which in itself is only the kernel.

Thanks for mentioning NuttX, I just tried it. I had something more similar to FreeRTOS in mind when I posted this topic, but NuttX seems different/almost like an OS and pretty interesting.

[metebalci]

The source code is there: https://github.com/weston-embedded/uC-OS3
It's usually better regarded for "safety-critical" applications than FreeRTOS. Again, haven't had a look at it in a while. But I'd say it's an alternative worth a look.

When the application requires a certification or compliance, is a commercial product a must ? So things like aerospace, automative and maybe medical are mostly covered by the commercial vendors ?

[eutectique]

When the application requires a certification or compliance, is a commercial product a must ? So things like aerospace, automative and maybe medical are mostly covered by the commercial vendors ?

If you want to certify your product (hardware and software) for safety and don't have a certificate for an RTOS yet, you would need to certify that RTOS as well.

Good luck.

[shtirka]

... because there's only a very limited number of use cases where [RTOA] truly shines (like when you need a highly deterministic system behaviour, or when you have strict demands on system performance or effective/maximum throughput) so using an RTOS will normally cause you more grief than it's worth (because usually it takes more effort to develop a solution to a problem or the solution becomes more complex than it needs to be - and also the system becomes more difficult to support in the long run).

I don't disagree, but I will highlight that if you need "highly deterministic system behaviour" and "strict demands on system performance" then you need to look at much more than the RTOS.

The best overall solutions I have seen for hard realtime performance guarantees are FPGAs and XMOS' xCORE processors running xC. The RTOS is, to all intents and purposes, implemented in hardware, the IDE tells you the min/max code execution times (none of this "measure and hope you have bumbled across the worst performance" rubbish!), USB and Ethernet libraries, 4000MIPS MCUs, buy them at DigiKey.

Example of hard realtime capabilities: software captures/generates 100MB/ Ethernet bitstreams or USB comms, performance guaranteed while the processor is also doing all the interesting things with hard realtime guarantees

Naturally, as will all technologies, there are swings and roundabouts. I have found the devices fun and easy to program with no late surprises due to, say, errata sheets. I had my first capture+process+USB comms to PC application running within a day of receiving the boards.

I will admit that I may have been perhaps a bit too categorical in my wording - perhaps I should have just said "deterministic behaviour" and "a bit higher effective throughput than you can achieve if you are sloppy" since in quite a few projects you still have quite a bit of leaniancy that would still allow the microcontroller+RTOS combination (where an RTOS may well also act as a bit of a "glue" or perhaps for managing the project complexity as well as increasing the performance) rather than going all in on it like in commercial/military aviation. So I guess this is part of the overall "context of applicability" of an RTOS - the part of the problem here being that the original question was a bit theoretical to begin with

Ilya

[tggzzz]

... because there's only a very limited number of use cases where [RTOA] truly shines (like when you need a highly deterministic system behaviour, or when you have strict demands on system performance or effective/maximum throughput) so using an RTOS will normally cause you more grief than it's worth (because usually it takes more effort to develop a solution to a problem or the solution becomes more complex than it needs to be - and also the system becomes more difficult to support in the long run).

I don't disagree, but I will highlight that if you need "highly deterministic system behaviour" and "strict demands on system performance" then you need to look at much more than the RTOS.

The best overall solutions I have seen for hard realtime performance guarantees are FPGAs and XMOS' xCORE processors running xC. The RTOS is, to all intents and purposes, implemented in hardware, the IDE tells you the min/max code execution times (none of this "measure and hope you have bumbled across the worst performance" rubbish!), USB and Ethernet libraries, 4000MIPS MCUs, buy them at DigiKey.

Example of hard realtime capabilities: software captures/generates 100MB/ Ethernet bitstreams or USB comms, performance guaranteed while the processor is also doing all the interesting things with hard realtime guarantees

Naturally, as will all technologies, there are swings and roundabouts. I have found the devices fun and easy to program with no late surprises due to, say, errata sheets. I had my first capture+process+USB comms to PC application running within a day of receiving the boards.

I will admit that I may have been perhaps a bit too categorical in my wording - perhaps I should have just said "deterministic behaviour" and "a bit higher effective throughput than you can achieve if you are sloppy" since in quite a few projects you still have quite a bit of leaniancy that would still allow the microcontroller+RTOS combination (where an RTOS may well also act as a bit of a "glue" or perhaps for managing the project complexity as well as increasing the performance) rather than going all in on it like in commercial/military aviation. So I guess this is part of the overall "context of applicability" of an RTOS - the part of the problem here being that the original question was a bit theoretical to begin with

Understood and accepted. I deliberately pushed the interpretation of the phrases beyond your intended meaning.

Nonetheless, it is useful to appreciate the limits and what can be achieved by using non-mainstream technology. If nothing else it concentrates the mind on what ordinary mainstream technologies doesn't achieve.

Whether it is worth pushing to achieve such limits is a separate issue to be decided on a case-by-case basis

[cv007]

Azure RTOS is/was licensed by manufacturers, such as stm32, microchip, nxp, renesas, and will soon become Eclipse ThreadX-

https://techcommunity.microsoft.com/t5/internet-of-things-blog/microsoft-contributes-azure-rtos-to-open-source/ba-p/3986318
https://threadx.io/

I briefly tried it on a Renesas RA series mcu and it seemed ok to me (meaningless). It is pretty lightweight, and was not hard to use. I also tried FileX and I guess it gives you an alternative to FatFS with maybe a better way to deal with exFat with respect to licenses.

[SiliconWizard]

A few people mentioned the peripheral support/stacks, which I had also in my mind. Not all, but I looked at some RTOSs, and it seems some of the peripherals are usually supported like network, but for example USB is not.

Yes, actually many network/radio stacks not only support RTOSs (often at least FreeRTOS), but often are not usable (at least without going through gigantic hoops) without one.
Which is a pretty common reason, from ME, for people to start using some RTOS. Just to be able to use a network or radio stack. And they build from there.

It's an "ecosystem" matter. Just like when people choose a programming language because of some available libraries, and then they build everything around it.

[shtirka]

... because there's only a very limited number of use cases where [RTOA] truly shines (like when you need a highly deterministic system behaviour, or when you have strict demands on system performance or effective/maximum throughput) so using an RTOS will normally cause you more grief than it's worth (because usually it takes more effort to develop a solution to a problem or the solution becomes more complex than it needs to be - and also the system becomes more difficult to support in the long run).

I don't disagree, but I will highlight that if you need "highly deterministic system behaviour" and "strict demands on system performance" then you need to look at much more than the RTOS.

The best overall solutions I have seen for hard realtime performance guarantees are FPGAs and XMOS' xCORE processors running xC. The RTOS is, to all intents and purposes, implemented in hardware, the IDE tells you the min/max code execution times (none of this "measure and hope you have bumbled across the worst performance" rubbish!), USB and Ethernet libraries, 4000MIPS MCUs, buy them at DigiKey.

Example of hard realtime capabilities: software captures/generates 100MB/ Ethernet bitstreams or USB comms, performance guaranteed while the processor is also doing all the interesting things with hard realtime guarantees

Naturally, as will all technologies, there are swings and roundabouts. I have found the devices fun and easy to program with no late surprises due to, say, errata sheets. I had my first capture+process+USB comms to PC application running within a day of receiving the boards.

I will admit that I may have been perhaps a bit too categorical in my wording - perhaps I should have just said "deterministic behaviour" and "a bit higher effective throughput than you can achieve if you are sloppy" since in quite a few projects you still have quite a bit of leaniancy that would still allow the microcontroller+RTOS combination (where an RTOS may well also act as a bit of a "glue" or perhaps for managing the project complexity as well as increasing the performance) rather than going all in on it like in commercial/military aviation. So I guess this is part of the overall "context of applicability" of an RTOS - the part of the problem here being that the original question was a bit theoretical to begin with

Understood and accepted. I deliberately pushed the interpretation of the phrases beyond your intended meaning.

Nonetheless, it is useful to appreciate the limits and what can be achieved by using non-mainstream technology. If nothing else it concentrates the mind on what ordinary mainstream technologies doesn't achieve.

Whether it is worth pushing to achieve such limits is a separate issue to be decided on a case-by-case basis

That's exactly it! And also one of the potential issues (this isnt something that I have personally come across because I dont mix C++ and RTOS concept - at least to the point of it becoming an issue) is the use of C++ exceptions in an RTOS (at least this is something that can potentially cause issues in FreeRTOS): apparently they can cause havoc with deterministic behaviour on an RTOS because of how compilers treat them (I dont remember the exact details of that, but I can certainly dig up the link to that discussion should you wish to). So you do need to be aware of not just of the context of applicability of an rtos as a tool but also of any potential issues in order to avoid a potential mess (which could be epic    )

Ilya

[Smokey]

I've always selected the RTOS because it was dictated by the built in vendor stack I had no control over anyway.
If I wanted the have the BLE functionality of the chip I selected I was going to use freeRTOS.  End of story. 

[tggzzz]

... because there's only a very limited number of use cases where [RTOA] truly shines (like when you need a highly deterministic system behaviour, or when you have strict demands on system performance or effective/maximum throughput) so using an RTOS will normally cause you more grief than it's worth (because usually it takes more effort to develop a solution to a problem or the solution becomes more complex than it needs to be - and also the system becomes more difficult to support in the long run).

I don't disagree, but I will highlight that if you need "highly deterministic system behaviour" and "strict demands on system performance" then you need to look at much more than the RTOS.

The best overall solutions I have seen for hard realtime performance guarantees are FPGAs and XMOS' xCORE processors running xC. The RTOS is, to all intents and purposes, implemented in hardware, the IDE tells you the min/max code execution times (none of this "measure and hope you have bumbled across the worst performance" rubbish!), USB and Ethernet libraries, 4000MIPS MCUs, buy them at DigiKey.

Example of hard realtime capabilities: software captures/generates 100MB/ Ethernet bitstreams or USB comms, performance guaranteed while the processor is also doing all the interesting things with hard realtime guarantees

Naturally, as will all technologies, there are swings and roundabouts. I have found the devices fun and easy to program with no late surprises due to, say, errata sheets. I had my first capture+process+USB comms to PC application running within a day of receiving the boards.

I will admit that I may have been perhaps a bit too categorical in my wording - perhaps I should have just said "deterministic behaviour" and "a bit higher effective throughput than you can achieve if you are sloppy" since in quite a few projects you still have quite a bit of leaniancy that would still allow the microcontroller+RTOS combination (where an RTOS may well also act as a bit of a "glue" or perhaps for managing the project complexity as well as increasing the performance) rather than going all in on it like in commercial/military aviation. So I guess this is part of the overall "context of applicability" of an RTOS - the part of the problem here being that the original question was a bit theoretical to begin with

Understood and accepted. I deliberately pushed the interpretation of the phrases beyond your intended meaning.

Nonetheless, it is useful to appreciate the limits and what can be achieved by using non-mainstream technology. If nothing else it concentrates the mind on what ordinary mainstream technologies doesn't achieve.

Whether it is worth pushing to achieve such limits is a separate issue to be decided on a case-by-case basis

That's exactly it! And also one of the potential issues (this isnt something that I have personally come across because I dont mix C++ and RTOS concept - at least to the point of it becoming an issue) is the use of C++ exceptions in an RTOS (at least this is something that can potentially cause issues in FreeRTOS): apparently they can cause havoc with deterministic behaviour on an RTOS because of how compilers treat them (I dont remember the exact details of that, but I can certainly dig up the link to that discussion should you wish to). So you do need to be aware of not just of the context of applicability of an rtos as a tool but also of any potential issues in order to avoid a potential mess (which could be epic    )

I played around with C++ in 1988, and quickly decided that it was more trouble than it was worth. The alternatives were better: C, Smalltalk[1], Objective-C, later Java.

For more "fun" with C++, the FQA (frequently questioned answers) is entertaining and very pointed. https://yosefk.com/c++fqa/ The C++ mob tend to squeal that it is out of date, but many.most of the points are still valid.

[1] yes, both Tektronix and HP built test equipment with embedded and/or headless Smalltalk C tended to be used for the "boring" low-level hardware interface.

[westfw]

is it common to make such a big change (change the architecture) during the lifetime of a normal product ?

Over the life of a single product?  Probably not.
Over the span of am emgineer's career at a company?  Yes.

Lots of companies transitioned from CISC to RISC processors, or to x86 as competing high-end CPUs died out.  (Consider Sun, or Apple.)
And lots of microcontroller-sized applications are now looking at the transition from 8bit to 32bit architectures (probably the only thing that makes an RTOS worth considering!)
Not that having to switch to a different programming paradigm isn't "good for you."  :-)

[metebalci]

Azure RTOS is/was licensed by manufacturers, such as stm32, microchip, nxp, renesas, and will soon become Eclipse ThreadX-

https://techcommunity.microsoft.com/t5/internet-of-things-blog/microsoft-contributes-azure-rtos-to-open-source/ba-p/3986318
https://threadx.io/

I briefly tried it on a Renesas RA series mcu and it seemed ok to me (meaningless). It is pretty lightweight, and was not hard to use. I also tried FileX and I guess it gives you an alternative to FatFS with maybe a better way to deal with exFat with respect to licenses.

This is a pretty big news.

[Siwastaja]

Yes, actually many network/radio stacks not only support RTOSs (often at least FreeRTOS), but often are not usable (at least without going through gigantic hoops) without one.

Yeah. External libraries which are written using linear code flows that assume multithreading is the #1 reason to use an OS which implements such threads. Own code can be written event-driven (e.g.: state change is an event) so that it runs in interrupt handlers, but modifying a large library to fundamentally different tyoe of impelemtation is a gigantic task, especially given that event-driven code is sometimes more tedious to write, although it offers inherent possibilities to log state changes etc. and as such in many ways is more understandable than many pieces of parallel linear flows that mutex on each other.

[SiliconWizard]

Yep. Network stacks are typical examples, but FatFs is another example (I remember having talked about it before). As all calls are blocking, if you're not using some kind of preemptive OS, it can be a royal pain to use without causing various issues elsewhere, or otherwise having to hack your way around it in very (IMO) ugly ways, such as calling your own "idle" function inside FatFs callbacks - yuck. No thanks. That's actually one reason I wrote my own (ex)FAT library for SD cards rather than use FatFs.

[tggzzz]

Yes, actually many network/radio stacks not only support RTOSs (often at least FreeRTOS), but often are not usable (at least without going through gigantic hoops) without one.

Yeah. External libraries which are written using linear code flows that assume multithreading is the #1 reason to use an OS which implements such threads. Own code can be written event-driven (e.g.: state change is an event) so that it runs in interrupt handlers, but modifying a large library to fundamentally different tyoe of impelemtation is a gigantic task, especially given that event-driven code is sometimes more tedious to write, although it offers inherent possibilities to log state changes etc. and as such in many ways is more understandable than many pieces of parallel linear flows that mutex on each other.

Exactly.

Newbies often don't recognise the differing characteristics and relative advantages/disadvantages of FSM vs "conventional linear" flow.

Choose the right tool for the job, and don't try to sharpen a hammer so it can be used with screws

[5U4GB]

In practice, implementing your own multitasking tends to become a chunk of spaghetti with subtle bugs and hacks, especially with multiple developers.

It's not so much the multitasking that adds value, its the device drivers and protocol stacks that come included.  I could quite happily live with little more than a single task, just a tight control loop that does everything, if in return I get someone else's device drivers and protocol stacks thrown in.

[5U4GB]

Not directly related to my question but is it common to make such a big change (change the architecture) during the lifetime of a normal product ?

Yes and no. You don't switch MCU families like you change underwear, but having the option to redesign into different MCU while having an idea that "oh, it will take two weeks to port" instead of "I have no idea whatsoever, maybe month, maybe a year!?" is a great safety net.

This was particularly apropos a year or two back when a response to "we want to order another 50K of xyz" would often be "sure, there's a 70-week lead time*** on that and you're #726 in the queue", at which point being able to switch to whatever happens to be available that doesn't have a 70-week lead time becomes a huge benefit.  I don't know how many redesigns I witnessed or was affected by at that time due to previously jellybean parts suddenly becoming unavailable, thus my earlier comment about the downsides but also the benefits if you need them of something like VxWorks.

*** It was always given as 52 weeks but never really was 52 weeks, I think that's just as far out as the unavailable-meter went.

[5U4GB]

When the application requires a certification or compliance, is a commercial product a must ? So things like aerospace, automative and maybe medical are mostly covered by the commercial vendors ?

For that you're going to be going with highly specialised RTOSes like ARINC653 ones or AUTOSAR stuff, alongside equally specialised development processes (have a look at the MISRA docs for an example of the automotive case) and running on top of equally specialised hardware, so unless you're at least a moderate-sized enterprise that's not really something you'll be playing with.

The one exception might be the FreeRTOS -> SafeRTOS upgrade path, but then getting that certified/approved is a lot more work than just retargeting your FreeRTOS project to SafeRTOS.

[metebalci]

When the application requires a certification or compliance, is a commercial product a must ? So things like aerospace, automative and maybe medical are mostly covered by the commercial vendors ?

For that you're going to be going with highly specialised RTOSes like ARINC653 ones or AUTOSAR stuff, alongside equally specialised development processes (have a look at the MISRA docs for an example of the automotive case) and running on top of equally specialised hardware, so unless you're at least a moderate-sized enterprise that's not really something you'll be playing with.

The one exception might be the FreeRTOS -> SafeRTOS upgrade path, but then getting that certified/approved is a lot more work than just retargeting your FreeRTOS project to SafeRTOS.

I know MISRA, but first time heard ARINC and AUTOSAR, thanks for the info.

When you say specialized hardware, do you mean redundant hardware components or something else ?

[5U4GB]

When you say specialized hardware, do you mean redundant hardware components or something else ?

I mean SIL/ASIL-compliant SoCs.  They're interrelated, SIL = safety integrity level, ASIL = automative safety integrity level, another equivalent being DO-178C design assurance levels for avionics.  So you'd be required to use, for example, an ASIL-D compliant SoC, and a full headache's worth of design assurance, for automotive systems like brake control where failures can be catastrophic, but only ASIL-B for things like headlights.  ASIL-B is also the point where the effort and cost required goes from annoying to painful, so staying within ASIL-B is a useful goal.

In practice this means things like dual cores in lockstep that cross-check each other, ECC on everything, lots of diagnostic and self-checking circuitry, isolation of functional units, massively overengineered clock and power subsystems, etc.  For an example, look at the TI TMS 570 SoCs, they're quite nice because you can buy their LaunchPad dev kits with an ASIL-D certified SoC for pocket change compared to what other devices will cost you.

[metebalci]

When you say specialized hardware, do you mean redundant hardware components or something else ?

I mean SIL/ASIL-compliant SoCs.  They're interrelated, SIL = safety integrity level, ASIL = automative safety integrity level, another equivalent being DO-178C design assurance levels for avionics.  So you'd be required to use, for example, an ASIL-D compliant SoC, and a full headache's worth of design assurance, for automotive systems like brake control where failures can be catastrophic, but only ASIL-B for things like headlights.  ASIL-B is also the point where the effort and cost required goes from annoying to painful, so staying within ASIL-B is a useful goal.

In practice this means things like dual cores in lockstep that cross-check each other, ECC on everything, lots of diagnostic and self-checking circuitry, isolation of functional units, massively overengineered clock and power subsystems, etc.  For an example, look at the TI TMS 570 SoCs, they're quite nice because you can buy their LaunchPad dev kits with an ASIL-D certified SoC for pocket change compared to what other devices will cost you.

Many thanks for the detailed answer.