What made you choose an RTOS, Linux, or both RTOS and Linux for a real product

[EVblog1]

Hello Experts,
 
I have about 2 years of experience as a test engineer in the automotive industry, and I'm trying to move into embedded software development.

For learning, I build small projects with LEDs, buttons, UART, SPI, I2C, Raspberry Pi, etc. I started with bare-metal, then FreeRTOS, and now I'm learning Embedded Linux. Since these are just learning projects, I can pick any approach I want, so I never have a real reason to choose one over another.

I've asked this question to others before, but I usually get answers like "use an RTOS if you have multiple time critical tasks" or "use Linux if you need networking." I understand those points, but I'm still not able to connect them to a real product.

Can you describe a situation from one of your projects where you or your team chose bare-metal, an RTOS, Embedded Linux, or both RTOS and Linux? You don't have to share any confidential details.

I'm more interested in what made you choose that approach.
For example:

• What problem were you trying to solve?

• What requirement made you say, "Yes, we need an RTOS," or "Linux is the better choice here" or both?

• Why wouldn't the other options work?

I'm asking because, as a hobbyist, I can make an LED blink with any of these approaches. In a real product, though, there must be practical reasons behind choosing one over another, and I'd like to understand that decision-making process from engineers who have built real products

[asmi]

You can actually have both - RTOS/baremetal latency with Linux functionality, as a lot of modern SoCs come with MCU core(s). So you handle latency-sensitive requirements on MCU core, and do the rest in Linux.

[nctnico]

You can actually have both - RTOS/baremetal latency with Linux functionality, as a lot of modern SoCs come with MCU core(s). So you handle latency-sensitive requirements on MCU core, and do the rest in Linux.

In addition: some (modern) multicore SOCs have virtualisation support. So you can run Linux and another OS on the same SOC.

[PlainName]

Can you describe a situation from one of your projects where you or your team chose bare-metal, an RTOS, Embedded Linux, or both RTOS and Linux? You don't have to share any confidential details.

One reason to use an RTOS, for non-realtime things, is simply to better partition the code. I think the term 'bare-metal' is misleading since you use it to mean 'non-RTOS', but what it should mean is that you implement the low level code that makes it run, not just the high level processes that do something.

Anyway, one non-RTOS project involved critical timing. Actual real-time, for a change! There were three main parts: reading an accelerometer and gyro, doing something with the data (basically making it sensible and setting a port pin), then sending the result to a remote device via a serial port and receiving commands. Usually you would have the serial stuff working under an interrupt, so you fill or read a buffer and the serial ISR deals with getting that buffer data in and out. But I couldn't afford for an interrupt to upset the timing of the main code, so for this project the serial handler was all done from the main loop.

A timer interrupt would trigger the accelerometer and gyro reading which would then continue to the data manipulation and port output, still under that same timer interrupt. The code had to complete before the next interrupt, obviously, but that wasn't a problem. The main loop would just deal with talking to the remote, deciding what to say and then sending it, at the lowest priority level. Worst case, that could just run out of time due to the accelerometer and gyro code hogging the processor and drop packets, but the system allowed for that. In this case an RTOS would not have been appropriate.

A different project, which just deals with turning the garage light on and off, had no hard timing requirements. It received switch and timer events and then sent light commands. There's also a web configurator, and because there was thus WiFi access, a command line processor. It could easily have been a single big loop, even with the wifi stuff, but using the RTOS means it's nicely partitioned and modular.

[EVblog1]

I intentionally didn't mention bare-metal because I thought it might take the discussion in a different direction.

I'm mainly interested in projects where you chose an RTOS, Embedded Linux, or a combination of both. I'd really like to understand what requirements or constraints led to that choice.

If you're able to share the reasoning behind that decision, it would really help me understand how these decisions are made in real products.

[PlainName]

I use an RTOS on typical embedded microprocessors. As I say, it helps modularise the software design.

I would use Linux when I need something chunky and powerful. The project where it was installed from source (that is, even the compiler built from source) was a 32-channel digial video recorder. Significant data throughput and several resource-intensive processes running in parallel - DMA, larger circular buffers, disk streaming, etc. Essentially a custom-built PC.

[Siwastaja]

Difference between "bare metal <-> RTOS" is much smaller than the difference between "RTOS <-> linux", since what most people mean with RTOS (typical example: FreeRTOS) is a very thin layer easy to poke holes at, and on the other hand, it also doesn't provide things most people expect from operating systems, like easy integration of complex drivers and networking stacks. So development for RTOS vs. bare metal is quite similar; development for linux is quite different.

The most notable thing RTOS gives you is parallelism in form of threads of linear code execution. In bare metal, you can use prioritized interrupts for very neat parallel FSMs with timing guarantees, but your functions start from the beginning of the function when triggered. It forces a different coding style entirely. "RTOS paradigms" are less real-time, though. RTOS may make integrating e.g. USB or TCP/IP stack easier (depending on which stack, and how the existing code is written), but it's still far from the ease-of-use you get with linux where USB and TCP/IP "just works".

So RTOS vs. bare metal is less significant decision than it sounds and more like a matter of taste. Also possible to refactor between using and not using one, after the fact.

HW is the driving factor, really. You use linux if you can afford the HW which conveniently runs it, say, you have the BOM budget in tens of $ and power budget in ~1W range, and need something like USB / networking (classic examples), large displays, something computing-heavy; i.e., want to leverage existing software in plug-and-play way.

You don't use linux when you want a very small, cheap, low-power solution, or don't need much existing large software pieces - we are talking about the $1, 100KB, tens of mW stuff. TCP/IP/networking is available for microcontrollers, too (like, lwip, mbedtls), so there's some overlap. There are also external offloading chip solutions, like the Wiznet "hardware" TCP/IP chips. But linux will be much more flexible, so it's kind of lower risk; it's easy to find people who can set up a linux on a small SBC and make it reliably do stuff like networking, USB and user interfaces.

The small microcontroller solution will be more difficult to achieve, with far fewer people available who can pull it off, but it has chances to boot faster, cost much less, and consume less power.

Our energy management IoT box is bare-metal microcontroller solution and we use Wiznet W6100 hardware TCP/IP / ethernet chip on it. And it is not plug-and-play, getting the Wiznet chip to work reliably is a task of its own; it's not a full hardware solution, and the provided example network library code is littered with bugs and security holes we have been fixing. We also have to live with fact that maybe 0.3% of customers suffer from connection reliability problems with Wiznet + some unlucky WiFi extender + some unulucky router, when all their other devices "mostly work". Embedded linux SBC would probably also "just work" with some random issues, but Wiznet can totally die, and the issue is undebuggable. Failure rate (maybe one in 300) is awkwardly in the transition area of "doesn't matter, can be ignored" and "causes colossal pain we must deal with". When one chooses linux, they won't be dealing with issues like this, but then maybe they are dealing with something else, like boot time. And unreliable drivers can be a thing even on linux, so always choose the HW wisely.

DFM, design for manufacturability usually favors the microcontroller solution, actually. If you are a large, serious player, you of course have resources to design your own computer that runs linux (something nctnico for example does). For a small player / hobbyist, designing an MCU board is a lot easier, and buying the $1 MCUs from Mouser/Digikey in tens of thousands units is trivial. You can't buy 10000 Raspberry Pis! If your product is based on a ready-made SBC because you were not able to design your own, check very carefully the availability and lead times of the SBC. Raspberry Pi has been sometimes completely unavailable in any quantity for months. That will kill your business if you depend on it.

[ejeffrey]

I've chosen embedded Linux for two main reasons.  The first is when I needed a powerful high performance networking e.g.: a high speed data acquisition system.  The second is just for easy administration and application deployment.  i.e., you can just use SSH to copy applications onto the filesystem.

For micorcontrollers, I've generally not chosen an RTOS but done bare metal.  In my case, these were simple enough that a simple hand written main loop plus a few interrupt handlers was enough for the realtime needs.  In many cases, the microcontroller is paired with a Linux host over a serial port.  In that case the smarts are pushed into the Linux OS, and only the low level tasks handled by the MCU, and for this I avoided needing to use an embedded RTOS.  The times where I looked at an RTOS I always ended up deciding that a two-processor system with a bare metal micro + raspberry pi or similar.

The constraint I haven't really had to deal with is cost.  All of these have been either personal projects where the unit cost isn't that important, or a small part of a low volume industrial product that are not sensitive to paying $100 for an SBC.  I haven't ever had to cost reduce an IoT product, or deploy millions of units of something.

[SiliconWizard]

Choosing Linux is rarely optional in practice: Linux requires a proper CPU to run on, and if you're using such a CPU, then using Linux is usually the least painful path, as the relevant SoCs rarely have official support for "baremetal" development (and if you want that, you're usually on your own, not that it's impossible). So it's a lot a matter of the processing power you need for your application, and if this processing power leads to a typical Cortex-A SoC or equivalent, then using Linux is usually the path that makes the most sense. Also, Linux has a wide ecosystem with support for storage, networking, USB, displays, ... so you get all that as a byproduct of choosing Linux.

Now, while you can compile a Linux kernel with RT patches (now officially supported in the kernel), it's rarely as "real-time" as what you could get on a MCU with no OS or some kind of RTOS. So it's not unusual to have both Linux (if you need the power/ecosystem) and a RTOS on a separate MCU or separate core in the same SoC.

As to RTOSs on MCUs, as others have said, they allow partitioning your firmware in a more systematic way and bring you a lot of niceties such as ready-to-use queues, sync primitives, and all that. But the biggest plus for many projects IMO is the fact that they are preemptive. Preemption makes it way, way easier to make the best out of CPU time when you need to handle storage and networking, for instance. You can of course do this without a preemptive OS at all, but then it requires making everything asynchronous and quickly becomes a big headache, especially since most existing libraries for filesystems (such as FatFs) or networking (such as lwIP) are not made to easily run on non-preemptive systems without a lot of work. With a preemptive scheduler, as long as you properly master synchronization concepts, it becomes much easier.

[Siwastaja]

For micorcontrollers, I've generally not chosen an RTOS but done bare metal.  In my case, these were simple enough that a simple hand written main loop plus a few interrupt handlers was enough for the realtime needs.

(emphasis mine), and something along this line is so commonly heard it deserves a rectification. Bare metal interrupt handler solution is not used as opposed to RTOS because it's "good enough"; the interrupt handler solution is absolutely the best real-time option and beats anything the RTOS offers by orders of magnitude (except, of course, using the exact same interrupts in the RTOS - they usually do not prevent that).

The cause of all the confusion is the perspective from which the "real time" qualifier is added to the base noun "operating system". RTOS is more real-time not compared to bare-metal, but when compared to a general-purpose operating systems, some of which (say, Windows NT) are not designed really to give much any kind of task deadline design patterns.

Why I find this rectification important, is because it totally changes the purpose of an RTOS.

The misunderstanding: RTOS is used on a microcontroller because it's realtime
Correct: RTOS is used on a microcontroller because it's an OS

RTOS gets chosen as the design pattern when some convenience of an OS - e.g., time-slice scheduling, threads of linear code with mutexes / semaphores - are desired; specifically this happens when having to depend on existing code (like networking or file system libraries) designed to run on a top of a typical OS (RT or not). But these mechanisms are significantly worse on real-time than any sensible base-metal solutions are. RTOS patterns are used for real-time systems when and only when they are good enough, not the other way around.

[nctnico]

Actually without having a very specific number as a time constraint, the term realtime is absolutely meaningless. For example: every PC can do the realtime task of playing video and audio. RTOS is more like a name for a minimalistic, small memory footprint OS.

[ejeffrey]

For micorcontrollers, I've generally not chosen an RTOS but done bare metal.  In my case, these were simple enough that a simple hand written main loop plus a few interrupt handlers was enough for the realtime needs.

(emphasis mine), and something along this line is so commonly heard it deserves a rectification. Bare metal interrupt handler solution is not used as opposed to RTOS because it's "good enough"; the interrupt handler solution is absolutely the best real-time option and beats anything the RTOS offers by orders of magnitude (except, of course, using the exact same interrupts in the RTOS - they usually do not prevent that).

It's not about more or less "real time" -- it's how complex you scheduling needs are.  For sure just putting your real-time code in interrupt handlers is the lowest and most deterministic latency.  The problem is when you need multiple long running tasks with different priorities and inter-task communication that still need real-time guarantees.  That's when you really "need" an RTOS (or you need to rethink your project...  this sort of system is really easy to f-up.)

But yes, another way to use an "RTOS" on an microcontroller is just to use it as an OS that happens to run on an MCU.  All of your realtime code goes in interrupt handlers, and the RTOS job is just to schedule background tasks and provide IO abstraction layers.  Like I said, I never really used a microcontroller RTOS, but all of the cases I considered it would have been like that.  And in my situation the solution was always just to offload the background tasks to a Linux OS, and then leave just the core logic on the MCU.

[nctnico]

For micorcontrollers, I've generally not chosen an RTOS but done bare metal.  In my case, these were simple enough that a simple hand written main loop plus a few interrupt handlers was enough for the realtime needs.

(emphasis mine), and something along this line is so commonly heard it deserves a rectification. Bare metal interrupt handler solution is not used as opposed to RTOS because it's "good enough"; the interrupt handler solution is absolutely the best real-time option and beats anything the RTOS offers by orders of magnitude (except, of course, using the exact same interrupts in the RTOS - they usually do not prevent that).

It's not about more or less "real time" -- it's how complex you scheduling needs are.  For sure just putting your real-time code in interrupt handlers is the lowest and most deterministic latency.  The problem is when you need multiple long running tasks with different priorities and inter-task communication that still need real-time guarantees.  That's when you really "need" an RTOS (or you need to rethink your project...  this sort of system is really easy to f-up.)

But yes, another way to use an "RTOS" on an microcontroller is just to use it as an OS that happens to run on an MCU.  All of your realtime code goes in interrupt handlers, and the RTOS job is just to schedule background tasks and provide IO abstraction layers.  Like I said, I never really used a microcontroller RTOS, but all of the cases I considered it would have been like that.  And in my situation the solution was always just to offload the background tasks to a Linux OS, and then leave just the core logic on the MCU.

Either way, if you have several timing critical tasks, the software requires careful design to meet all the (time) deadlines. My approach is typically to share as much of the time critical stuff in one or two fixed frequency interrupts and use software to do things that need to be done at longer intervals instead of having many seperate interrupts.

One of the problems with having several OS threads is that meeting time deadlines becomes harder because the execution paths become harder to predict. Especially if you throw a filesystem and storage into the mix. Interestingly having many OS threads and many interrupts are posing the same problem. I like to model the interrupt handler as a hardware pre-emptive time slicer (like an OS does).

[PlainName]

Choosing Linux is rarely optional in practice: Linux requires a proper CPU to run on, and if you're using such a CPU, then using Linux is usually the least painful path, as the relevant SoCs rarely have official support for "baremetal" development (and if you want that, you're usually on your own, not that it's impossible).

I think that condenses the reason for Linux quite nicely

[EVblog1]

A couple of examples from my experience:

• A small wearable device with BLE and Cellular module: The application MCU was a small popular model with RTOS. I had to use an MCU to achieve the required battery life. The cellular module that I occasionally need to turn on runs Android internally, although I treat it as a black box and talk to it over serial port.

Thanks for sharing the example.

I'm curious about the timing requirements on the MCU. Roughly how many time-critical tasks did it have, and what kind of deadlines were they expected to meet? I  mean deadline time  about microseconds, 10–20 ms, 50 ms, or even longer?

Also, what would happen if  those tasks missed its deadline? Would it just affect performance or battery life?

I'm trying to relate the real-time requirements to the decision to use an RTOS.

[EVblog1]

I think this is where I'm getting confused, so let me ask it a different way.

Suppose you're designing a system that has a few normal tasks and a few time-critical tasks that must complete before their deadlines. If a deadline is missed, the system may not behave correctly.

Now imagine you have two options: implement the whole system using bare-metal with interrupts, or use an RTOS like FreeRTOS.

My current understanding is that if the timing requirements can already be met with a bare-metal design, then that would seem like the simpler solution. It avoids the scheduler, context switching, extra RAM usage, and the additional code that comes with an RTOS. Because of that, I'd expect it to have less overhead and lower latency.

This is the part I don't understand.

If the bare-metal implementation can already meet all the deadlines, what makes you decide that an RTOS is still the better choice? What changes in the system that makes you think, "At this point, an RTOS is the right tool"?

I'm asking because this is something I genuinely struggle to understand. I know many real products use an RTOS, so I'm clearly missing part of the picture, and I'd like to understand what that is.

[uer166]

What you described is a false dichotomy.

A well designed system in your example could be "both" bare metal and RTOS.

Specific example for Arm + FreeRTOS:

The critical stuff runs as interrupts using ARM NVIC (so they can nest if need be according to their deadlines).

The non-critical stuff may be FreeRTOS tasks, and they can communicate data back and forth to ISRs using either the primitives provided to you by the FreeRTOS kernel itself (e.g. _FromISR API for FIFOs), or your own primitives.

The tasks provide you task-switching so you can write long tedious code without worrying about deadlines or cooperation (e.g. some GUI rendering code), while the ISRs can be used to provide timing and/or jitter guarantees for some peripheral driver or whatever.

[uer166]

Side note: I pretty much never use RTOS even for non-critical tasks because nested interrupts on ARM provide excellent hardware task switching as well. You do need to have some kind of atomic ways to transfer data that is reliable. The most trivial way is 32-bit aligned self-contained data (e.g. floats) on a 32-bit in-order CPU core.

[5U4GB]

A couple of examples from my experience:
Linux Cons: Larger power consumption, physically large SoM or your own custom PCB which is quite a job. Setting up a new build can be a huge task - especially if vendor BSP is bad.

Another major Linux con: Lots and lots and lots of things to fail if two processes get out of sync or some daemon's config gets slightly messed up or something fills up all the swap space or someone gets file locking wrong or ...

If you've just got a control loop that does A, B, C, D and then back to A again then there's vastly less that can go wrong.

A few other things:

• Security-wise the majority of your problems will go away if you're running something that an attacker can't get a shell on because there isn't one.
• Any cowboy thinks they can write stuff for Linux (see above). For an RTOS or bare-metal you actually need to know what you're doing. Downside is that people like this are in much shorter supply than cowboys.

[Siwastaja]

It's not about more or less "real time" -- it's how complex you scheduling needs are.  For sure just putting your real-time code in interrupt handlers is the lowest and most deterministic latency.  The problem is when you need multiple long running tasks with different priorities and inter-task communication that still need real-time guarantees.  That's when you really "need" an RTOS (or you need to rethink your project...  this sort of system is really easy to f-up.)

To be more exact: RTOS offers tools for long-running compute-intensive (relatively speaking, for a small MCU even some simple algorithm can be compute-intensive) tasks to time-share more easily.

For hard deadlines and predictable reaction to input signals, you would use the exact same interrupt handlers you use on bare metal, on the RTOS. But RTOS may make the variable-timing background tasks easier to organize, as parallel linear code flows that flow as quickly as they can, with synchronization barriers between them when they depend on each other. But the opposite of what people seem to expect is true: RTOS won't magically be able to guarantee that algorithm finishes in time; it's all about the algorithm itself, and locking (like, mutexes between threads; priority inversion) is something the programmer still needs to solve, and the problem is exactly this: you have asynchronously running processing; you made it run asynchronously because it's easier so that you don't need to think when it runs, but then you added synchronization against other such asynchronous processing - which you also chose to do asynchronously because of ease. And this is the blind spot. It's exactly what leads you missing deadlines in 0.001% of cases or get a priority inversion and get your code crash on Mars.

A bare-metal approach might force you to dissect your algorithm into smaller chunks and prove that each individually fits in your available tick, and prove the timeframe in which things happen (how the "threads" synchronize). The asynchronous piece becomes smaller. This is more work, but it also gives you a real worst-case performance guarantee; understanding into the system. The advantage of an RTOS would be that you are able to run many of such processes in parallel, so that they average out, which means that your average timing case improves, probably your P99.9 case improves, but you just created an edge case which is difficult to test. In other words, if your algorithm consists of 10 sequential steps all taking something between 500-1500ms, averaging out to, say 900ms, with an RTOS you can fit the linear code into 9 seconds of average runtime, and in bench testing you find out it never exceeds 10 seconds, until one day it does, since one of the steps took longer and the next step wasn't shorter like it usually is. In bare metal solution, you divided that into 20 chunks because those 1500ms chunks didn't fit in your 1-second tick, so now you are spending more time, but now every step has that extra slack. The difference is forced visibility into smaller parts: you logged the worst case over every step in isolation because you had to, it was obvious you need to do it. The manually scheduled bare metal solution was, surprisingly, simpler to reason about and you know it takes exactly 20 seconds worst case, best case, average case.

If it feels very difficult, it's a warning sign that your system is complex! You may succeed better with RTOS, but the red flag remains: your system will very likely contain some serious corner case timing bug.

[5U4GB]

There's also the other side of the bare-metal approach, if the thing you're working with implements the equivalent of a dozen lines of ladder logic and has a SoC clocked at 48MHz, so it's spending 99.9% of its time in a sleep state, you probably don't need an RTOS.

Now imagine setting up a Linux device to do that...

[EVblog1]

I think it would help the discussion if, instead of general statements, you could refer to one specific project where you chose an RTOS and explain what requirements justified that decision. I find it much easier to understand the reasoning when it's tied to a real example, and I think we can have a more meaningful discussion from there.

[5U4GB]

I was just about to suggest the opposite, give some examples of what you'd like to do, because otherwise the most you'll get is rather general advice.

In my case the example of a 48MHz SoC was close to a real thing, sample a bunch of inputs, perform some actions, trigger some outputs if required. Since it was software there's a bit more functionality there than you could do in basic ladder logic, but it was a similar sort of setup.  All bare-metal, I would have probably gone with an RTOS because it makes things a bit cleaner than working at the bare-metal level, but that was someone else's decision.  It also meant that you could recover from errors via rejuvenation in less time than the scan interval because the restart time was a few thousand clock cycles.

[Siwastaja]

Actually without having a very specific number as a time constraint, the term realtime is absolutely meaningless. For example: every PC can do the realtime task of playing video and audio.

Actually two numbers: time constraint and probability of missing that (or, worst-case time constraint). Every PC can play video and audio, but they don't guarantee they wouldn't sometimes drop a frame. Most consumers of video never notice it at all; some notice but don't care. Heck, even crappy 24<->25 FPS conversions by entirely dropping or duplicating frames, which produces annoying jerk of motion repeatedly every second, do not bother many normies at all.

But if missing the deadline even once means your thing crashes on to the surface of Mars and is dispersed into small droplets of molten metal, ruining your $100 000 000 000 000 000 billion project, then you don't want to use Windows NT kernel on it, even though it manages stutterless video delivery most of the time. Then again, FreeRTOS is magically not going to prove anything, either.

[SpacedCowboy]

Seems like I can't post anything of a reasonable size to the forum right now - maybe a limit of 512 bytes or so... Guess I'll report that.