What made you choose an RTOS, Linux, or both RTOS and Linux for a real product

[SiliconWizard]

I think it would help the discussion if, instead of general statements, you could refer to one specific project where you chose an RTOS and explain what requirements justified that decision. I find it much easier to understand the reasoning when it's tied to a real example, and I think we can have a more meaningful discussion from there.

So, you're not just bombarding the forum with questions, but also directing the way people should answer them.

[nctnico]

Actually without having a very specific number as a time constraint, the term realtime is absolutely meaningless. For example: every PC can do the realtime task of playing video and audio.

Actually two numbers: time constraint and probability of missing that (or, worst-case time constraint). Every PC can play video and audio, but they don't guarantee they wouldn't sometimes drop a frame. Most consumers of video never notice it at all; some notice but don't care. Heck, even crappy 24<->25 FPS conversions by entirely dropping or duplicating frames, which produces annoying jerk of motion repeatedly every second, do not bother many normies at all.

But if missing the deadline even once means your thing crashes on to the surface of Mars and is dispersed into small droplets of molten metal, ruining your $100 000 000 000 000 000 billion project, then you don't want to use Windows NT kernel on it, even though it manages stutterless video delivery most of the time. Then again, FreeRTOS is magically not going to prove anything, either.

Yeah, but in such cases you may want to reconsider the design if there is absolutely no fault tolerance. IIRC Apollo 11 had computer problems caused by rogue interrupts during the descent onto the moon but it still managed to stay functional.

[ejeffrey]

Suppose you're designing a system that has a few normal tasks and a few time-critical tasks that must complete before their deadlines. If a deadline is missed, the system may not behave correctly.

Now imagine you have two options: implement the whole system using bare-metal with interrupts, or use an RTOS like FreeRTOS.

My current understanding is that if the timing requirements can already be met with a bare-metal design, then that would seem like the simpler solution. It avoids the scheduler, context switching, extra RAM usage, and the additional code that comes with an RTOS. Because of that, I'd expect it to have less overhead and lower latency.

This is the part I don't understand.

If the bare-metal implementation can already meet all the deadlines, what makes you decide that an RTOS is still the better choice? What changes in the system that makes you think, "At this point, an RTOS is the right tool"?

This goes back to: an RTOS is an operating system.  You would use it if you needed operating system features.  If you can put everything in interrupt handlers, that's going to be the lowest latency and may be the simplest.  But interrupt handlers should not sleep or block so they can't acquire a lock, perform many kinds of IO, or execute long-running computations without starving other work.

Here are some concrete if somewhat contrived examples where you might use an RTOS for it's real-time scheduling capability:

1: You have a single IO port (like an automotive CAN bus) that can receive messages that can be high or low priority.  You don't want to handle the message processing in the interrupt context because then the low priority work can't be interrupted.  So instead you write a task for each message type, and the tasks just sleep waiting for a message.  The interrupt handler's job is to receive the message, stores it, and wake the appropriate task. 

2: You have a high priority task (like say, controlling the motor speed in a quadcopter) but the control law depends on temperature and atmospheric pressure which are slowly changing and updated by a low priority background task.  The environmental data block is protected by a mutex.  The control loop can't be in an interrupt handler because it can't acquire the mutex.  So you make it an RTOS task.  Now importantly, when the background task is writing the environmental data block, it becomes high priority as it needs to finish the update and release the mutex before the control loop can execute.  An RTOS will handle this to prevent priority inversion.

Note: this is an undesirable design pattern.  In this example it might be better to arrange for the high priority task to work with slightly stale data rather than to use a mutex.  But for various reasons it is still common to end up needing to share locks between high and low priority tasks.

3: You have a bunch of periodic jobs that need to be done with different intervals, but you only have one hardware timer peripheral.  So you can't dedicate a timer to each job.  So you make each one a task, and have a single timer interrupt that is responsible for figuring out which tasks need to be waked, when the next timer will come due, and reprogramming the hardware timer.

There are other reasons to us an RTOS.  It may come with a lot of useful IO drivers, and it can be useful to have preemptinve multitasking even for non-realtime tasks such as a networking stack or filesystem interface.  A lot of people use an RTOS for these tools even if they don't need the realtime scheduling.  In that case, it's just a way to reuse existing code.

[nctnico]

Suppose you're designing a system that has a few normal tasks and a few time-critical tasks that must complete before their deadlines. If a deadline is missed, the system may not behave correctly.

Now imagine you have two options: implement the whole system using bare-metal with interrupts, or use an RTOS like FreeRTOS.

My current understanding is that if the timing requirements can already be met with a bare-metal design, then that would seem like the simpler solution. It avoids the scheduler, context switching, extra RAM usage, and the additional code that comes with an RTOS. Because of that, I'd expect it to have less overhead and lower latency.

This is the part I don't understand.

If the bare-metal implementation can already meet all the deadlines, what makes you decide that an RTOS is still the better choice? What changes in the system that makes you think, "At this point, an RTOS is the right tool"?

This goes back to: an RTOS is an operating system.  You would use it if you needed operating system features.  If you can put everything in interrupt handlers, that's going to be the lowest latency and may be the simplest.  But interrupt handlers should not sleep or block so they can't acquire a lock, perform many kinds of IO, or execute long-running computations without starving other work.

The latter is exactly the issue because it leads to the question: when and how are you going to synchronise getting the work done in time before the next interrupt fires. This is where the 'keep interrupts short' dogma fails because it only moves a problem into a different area without actually solving it (more likely making the problem bigger/worse). The real solution is to plan and divide processing time across the tasks while the system is being designed. From the design you know the time constraints and from there you can plan how and when pieces of code for various tasks can be executed.

For signal processing applications I typically have all the signal processing inside an interrupt. In some cases this interrupt can consume 90% of the processor time. Interrupts which are short by nature (like handling UART and USB) can interrupt the signal processing interrupt to do their thing. While the main tasks consumes the remaining processor time. Such a system is carefully planned where it comes to what is executed when.

[asmi]

I prefer using RTOS given the choice because it allows to use more event-driven approach to development, which is what is used in "big" software projects. Also designing a system architecture for RTOS forces you to think in terms of isolated sub-tasks and the way data flow between them, which somewhat decouples tasks from each other and allow parallel and independent development once the dataflow pattern is established.

[Smokey]

I essentially only use RTOS if some hardware support blob/driver requires it like BLE or WIFI.  If at all possible I try to keep microcontroller projects small and specific purpose and interrupt driven.  The code is more straightforward and much more deterministic. 

Start small and see what the minimum viable processor/system that would work and work your way up from the bottom.  Don't just start with embedded Linux for everything. 

This is assuming you plan on making a bunch of something.  If you are just making one of something for yourself then do whatever you are familiar with.

[EVblog1]

Based on everything I've read so far, this is the conclusion I've come to.

It seems to me that, for many real-time systems, choosing between bare-metal and an RTOS is often an engineering decision. It looks like both approaches can be used to build a real-time application, and it's not necessarily true that you must use FreeRTOS or another RTOS.

Right now, the biggest difference I see is in the programming style. An RTOS seems to become more useful as the system grows in size and complexity because it lets you organize the application into separate tasks, making the code easier to manage and maintain.

As for Embedded Linux, that part makes more sense to me. If the application needs OS features like a networking stack, file system, process management, USB, etc., then using Linux seems like the natural choice instead of implementing everything yourself.

So that's my current understanding. If I've misunderstood something or missed an important point, I'd really appreciate being corrected. Otherwise, if most of you agree with this conclusion, I think my question has been answered and we can probably end the discussion here.

[SpacedCowboy]

My old boss used to describe bare-metal programming as BLT - “Big Loop Technology”, where most things happened periodically and you used interrupts for things that didn’t. If everything used interrupts, your loop had wait-for-interrupt as its last statement…

Personally I have a project (now I’m retired) where I wanted all the gubbins that a Linux system would provide, but I also wanted that “instant-on” feeling, not to have to wait for a system to boot up through all the layers that Linux requires. The idea is to emulate the computers from my youth on a device I could plug in and use - in this case an Atari XL and an Atari ST, switchable as I saw fit at time of use. Goals were:

• “1 second to ‘ready’”. There’s something visceral about “switch on and go” that made the old computers much more like devices than modern “switch on and boot” ones
• It should be trivially field-upgradeable, so booting from SD was preferred (just put the SD card in a PC and copy the new boot-system onto it).
• It should be able to load and run binaries from SD
• It needs to drive a 1080p display in RGBA32 format
• It needs to have mouse, keyboard and general USB interfaces
• It should support networking

The tension here is that the first requirement (which I wasn’t willing to compromise on) is pretty much ruling out the system (Linux) that would make the rest pretty easy. Add that to the fact that I’ve always sort of wanted to have my own OS, and I chose to extend FreeRTOS since it's a much simpler system, although too limited for the above list as-is. The board I’m targeting is a Zynq 7020 MyIR Z-turn (at least for development) so the CPU is a dual-core ARM A9 at 866MHz - not ancient, but not modern either, and there’s all that nice programmable logic for the emulator…

In the manner of personal projects that don’t have deadlines, this has grown somewhat in scope over time. I now have:

• FPGA implementation of the 6502 (twice, actually, once for performance at 100MHz, once for fidelity at 1.79... MHz with the rest of the atari chipset)
  
• Shared objects and shared libraries
• Dynamic linking so programs can be loaded from SD, resolve against the shared libraries, and run as FreeRTOS processes
• Memory protection, fully implemented so we get VM, CoW etc.
• Boots to a graphical ready-to-use state in 2.5 secs, but 1.5 of that is my monitor syncing to 1080p, console is ~1 sec
• Virtual filesystem support, /tmp is RAM, /fujinet is network-mounted, elsewhere is SD
• /dev, /proc and a shell, network access to shell provided by sshd
• Kernel-based ring buffer for 'dmesg' logging and review
• Networking, and a BSD sockets interface, SNTP to get the system time, DHCP for network IP etc
• /dev/blitter for an 800MB/sec scaling, bilinear approximating, affine transform capable blitter that understands alpha and can blit via operation
• Implements a modern true-colour variant of GEM as its fundamental graphics system
• gemd runs at boot, then desktop.app is run (gem client)
• gemd owns the frame store, runs in retained mode, each window has backing memory, shared as a ‘virtual workstation’ (VW, basically a graphics context)to client apps
• Client can write to its own VW without IPC, then message gemd that region X is damaged, gemd repaints from bitmap
• Because every graphical surface is a VW, blitter can be used. It has a 1024-deep command queue and you can use it asynchronously or wait for your blit to complete
• Text is supported via Freetype, which renders to a system-wide glyph-cache of 64MB (IIRC) as needed using LRU to keep space available
• I always liked Objective C as the sweet spot for language complexity vs expressiveness, so there’s a compiler (the website is out of date, but it gives a feel of the language) … which is self-hosting. It actually targets a variety of back-ends (6502, m68k, A9 for the “on-board” devices, and Mac, Linux and Windows binaries to make development easier)
• You get the typical nice things from ObjC - automatic memory management with ARC, objects, protocols, the Foundation class library (mostly, not everything is ported).
• Theres also the XG framework instead of AppKit, which provides a very AppKit like interface (XGScrollView instead of NSScrollView, the same for TableView, ListView, CollectionView, and all the smaller button-type widgets. These render using the native toolkit on each platform, and via GEM on the board. It’s possible to develop on the Mac (say) and then compile the same source and see it on the board (or Windows, or Linux)

It’s not “done” yet, just got to the point where the compiler is self-hosting (writing the compiler itself in “xt”) and having any platform be able to generate code for any other without needing host-local compiler tools. Now that’s done, I have threads to implement across the language targets (mainly pthreads, so not hard), and then I want to tackle WebKit, so there’s a native browser-type widget/app.

At that point, I’ll probably open source it, there aren’t that many instances of expanding FreeRTOS to this extent out there, at least that I know of. I've written an OS before, but I wouldn't have tackled something as large as this without AI help along the way, so that whole thing was pretty timely from my perspective
 
So sometimes it’s worth it to do it all yourself, it just depends on what you want, really

[nctnico]

Right now, the biggest difference I see is in the programming style. An RTOS seems to become more useful as the system grows in size and complexity because it lets you organize the application into separate tasks, making the code easier to manage and maintain.

Not necessarily. Actually the opposite is true. In an embedded system most of the tasks are related. Seperating into more tasks means maintaining more relationships. This is a pitfall programmers tend to fall in.

Where the use of an RTOS comes in handy is when you have several big tasks which run mostly independently with very clearly defined interfaces. Like a network stack, http server, BLE stack, etc.

[paulca]

Money.

Start asking about the costs and benefits.  There you will find 90% of your answer.

The 10% where engineers get a choice or a say in the matter, its usually convenience which translates to speed and easy of development which gets picked.

The options have very different "footprints" and costs.

If you can do the job with a $0.35 MCU in 4kb of RAM why on earth would you budget something capable of running a BLE stack let alone a full TCP/IP stack?

[paulca]

Right now, the biggest difference I see is in the programming style. An RTOS seems to become more useful as the system grows in size and complexity because it lets you organize the application into separate tasks, making the code easier to manage and maintain.

Not necessarily. Actually the opposite is true. In an embedded system most of the tasks are related. Seperating into more tasks means maintaining more relationships. This is a pitfall programmers tend to fall in.

Im not sure separations of concerns and implementation behind an interface call be called "Pit falls".

The "pit falls" are in doing it wrong.

If you want a lesson in how to "decouple" code corrected the litmus test is simple.  Take every single c file in your project and write a "Unit test" for it.

If you can't you have a heavily coupled system and changing anyone part, changes all of it.  If your project is small enough for that not to hurt later, great.  Don't over engineer it.  But when it is.... dont come crying.

Defining the interface between components formally is also, hardly a pitfall.  Allowing them to grow organically without oversight is far more common in embedded in my eyes.

"Unit testing" is the search light for coupled code.  The moment you try and "instantiate" your module and it tries to access an external item, like an MCU register, FAIL.  Cannot be tested in isolation, can only be tested with external dependancies, so are you testing the code or the dependency?

"Process isolation" isn't really a thing in embedded though.  So I would ask ... why do you want the execution separation?  Thats a different thing to the code interface/component level separation.

In a "general purpose" big iron with virtual addressing splitting at the "Process" boundary and not the "Thread" boundary forces isolation, forces a clean interface and prevents sneaky "I'll just reference this other modules stuff for convenience".

In an RTOS, unless I am mistaken, there is nothing preventing TaskA from pilferring state from TaskB while nobody expected it.

The only reasons I can think of then is "cadence" and "idle states".  If you have two (or more) tasks with different cadence (interrupt rate, duty cycle etc.) or you have tasks which will take considerable time and want other things to still work.

Basically RTOS is a framework for writing super loops (aka roll your own threading).  Not much more.

Horizontal scaling is probably the other.  Processing multiple things in parallel usually benefits from such a task orchestration framework.

[NorthGuy]

An RTOS seems to become more useful as the system grows in size and complexity because it lets you organize the application into separate tasks, making the code easier to manage and maintain.

Not really. RTOS gives you threads. This lets you create several tasks and gives you an illusion that they all run simultaneously. RTOS does so by periodically stopping tasks and switching between them. This is called preemptive multitasking. That's all there is to it.

You pay a price for the "convenience" with a small performance hit, timing unpredictability, and necessity for synchronization primitives, such as mutexes.

This has nothing to do with complexity. Say, Windows kernel is much more complex than most of the embedded things, yet the majority of driver's code runs on so called Dispatch level where there's no threads and hence the reaction time is much better. This does not eliminate simultaneous execution because there are many CPUs which all run in parallel, but the absence of threads lets you replace heavyweight mutexes with speedy spinlocks.

[EVblog1]

So, if I understand your point correctly, are you saying that one of the main reasons peoples choose something like FreeRTOS is to save development time by using a well-tested third-party RTOS instead of building everything themselves? Instead of implementing your own scheduler, timers, synchronization primitives, and other infrastructure, you can rely on the RTOS and focus on the application itself.

In other words, if FreeRTOS is integrated and used correctly, it can reduce development effort even when a bare-metal implementation could technically meet the timing requirements. Is that basically the point you're making?

[KE5FX]

So, you're not just bombarding the forum with questions, but also directing the way people should answer them.

   To be fair, the clanker does seem to have inspired a thoughtful, informative discussion.  I look forward to querying it in a prompt someday when I am no longer able to avoid incorporating embedded Linux in a device.

[5U4GB]

If you can do the job with a $0.35 MCU in 4kb of RAM why on earth would you budget something capable of running a BLE stack let alone a full TCP/IP stack?

Because then you can call it "smart", charge ten times as much for it, have it phone home to your cloud infrastructure which you can then monetise, and more people will buy it because of the "features".  This engineering process is known under the technical term "enshittification".

[Siwastaja]

So, if I understand your point correctly, are you saying that one of the main reasons peoples choose something like FreeRTOS is to save development time by using a well-tested third-party RTOS instead of building everything themselves? Instead of implementing your own scheduler, timers, synchronization primitives, and other infrastructure, you can rely on the RTOS and focus on the application itself.

This is a common argument indeed, but of course it isn't true. No one is implementing their own OS (except those who actually are for whatever reason, but let's now ignore that special case).

OS might implement a generic tick that stores CPU state to allow multi-threading / time-slicing of arbitrary code. You would not replicate that. Your "scheduler" might be a lower-priority interrupt handler which does something every 1ms or 1s or whatever. But most importantly, your "scheduler" would be your business logic. You are concentrating on your application, all the time.

Synchronization primitives? disable_irq(...); ... enable_irq(...) are one-liners and come from manufacturer's existing header file. Calling it "implementing your own" is a huge overstatement. It is exactly same amount of work than using OS synchronization primitives, say, mutexes, except in bare-metal it is easier to understand what's actually going on under the hood.

You have fallen into the trap of magical thinking. The biggest problem with RTOS is that beginners are told to use them, and that they are more complicated while they eventually do simple things, but they abstract and hide what they are actually doing. This is the recipe for hard to find corner case bugs - timing misses, race conditions, priority inversion, deadlock. RTOS solves none of them. It offers tools to deal with them, but you need to be able to use those tools.

[NorthGuy]

In other words, if FreeRTOS is integrated and used correctly, it can reduce development effort even when a bare-metal implementation could technically meet the timing requirements. Is that basically the point you're making?

I wouldn't say this is true. By using RTOS, you're creating immensely different product compared to what it would be if you didn't use it.

Apparently by "timing requirements" you mean something totally different from what I do.

[paulca]

If you can do the job with a $0.35 MCU in 4kb of RAM why on earth would you budget something capable of running a BLE stack let alone a full TCP/IP stack?

Because then you can call it "smart", charge ten times as much for it, have it phone home to your cloud infrastructure which you can then monetise, and more people will buy it because of the "features".  This engineering process is known under the technical term "enshittification".

True.  Or go out of business trying.

EDIT: To cross-advocate though.  As a hobbiest I move towards "conflated platforms".  So I "pick" one example for each "granularity of task".  Below those floors I leave $$ on the BOM which would be removed in a commercial outfit.
If it can done with an arduino, why not.
It if needs a bit more stick with the STM32F411, all rounder.
Need BLE or Wifi?  ESP32.
Need grunt, clock, compute or seriously good periphs... STM32H7.

Often it's not just one of them, but a team.  ESP32 does the Wifi.  STM drives the screen, arduino the test rig.  That kind of stuff.

Its not practical to "pick one MCU" and stick with it for all projects.

[Fire Doger]

You are approaching them as "right and wrong ways", in the real world there is no right and wrong choices, only pros and cons....

In general, usually, RTOS is better choice when your superloop starts to look like an RTOS... but this doesn't mean that baremetal is a wrong choice, depends on your knowledge, how you implement the baremetal and other factors...

Similarly, this also applies for linux, believe it or not. And I can prove it with an example.

One competitor had a device with LCD, animation, graphics, etc running on Linux, arguably the standard of the market at the time.
My then company did the same device with freertos and TouchGFX.
Our device had less animation capabilities, it had stupid bugs, it took much much longer to fix bugs because we had to manage everything manually, and many more...
But as a company we took a huge portion of the market share because it was cheaper, and we had stock during covid...

The best advice is to know them equally if possible so that you can be agile and choose based on pros n cons of each implementation.

Currently for example I work on a design that linux might be a better fit, not game changing, but with more benefits...
But because I don't feel comfortable producing tens of thousands of devices that need to work continuously for the next 10-20 years in linux, I choose to go with FreeRTOS, with an abstracted core to be able to port it in the future if needed.

[5U4GB]

Need BLE or Wifi?  ESP32.

I think a more common one there is "Need to tie it into Home Assistant?  ESP32".

Or, alternatively, one of the million other works-with-ESP32 systems.

Apart from that, FreeRTOS + STM32 is a good combination for small embedded things.  The main benefit I find in an RTOS is that it's a HAL, I don't have to redo lots of things if I move to a slightly different SoC variant, or figure out which versions of hardware bugs I need to work around on this stepping.  Even if you don't necessarily need an RTOS, you can always use a HAL.

[peter-h]

The main benefit of an RTOS is that each task can be written as if it owns the machine.

This drastically simplifies programming. You don't need funny loops, horrid state machines which nobody can understand, etc.

Obviously things need to be done carefully if sharing data (mutex locks etc) but overall there is a huge benefit.

I've just done wifi using the WFM200S. Took a day (with Claude) to get it done. No need for ESP32 and the "Chinese risk".

[paulca]

The main benefit of an RTOS is that each task can be written as if it owns the machine.

Really?  I think that is false confidence.  You haven't bitten yourself yet.

[Siwastaja]

The main benefit of an RTOS is that each task can be written as if it owns the machine.

Not at all. The requirements to manage resource sharing is exactly the same, RTOS or bare metal; just the exact "how" is slightly different.

If you just "don't care", you will create the exact same hard-to-reproduce corner case bugs, both RTOS and bare metal.

The old wisdom, "parallel programming / multithreading is difficult", is true and there is no shortcut, really.

For sharing machine resources or data between threads of execution (and interrupts count as one), you need locks, mutexes, semaphores (as a concept; they can go under a different name, like "disabling interrupts") - and once you have those, you have conceptual risk of deadlock or priority inversion, and of course, missing deadlines you cared about. Between RTOS vs. bare metal, the structure how the issues are dealt with change somewhat, concepts themselves do not. For some, the OS abstractions may be easier to understand; for others, bare metal is more obvious - but as you demonstrate yourself, with RTOS there apparently lies a huge risk of magical thinking; false sense of security - people mistakenly think it solves more than it actually does.

And if one struggles to get a bare-metal software work reliably, and tries to apply RTOS as a fix, they will be disappointed. To write robust code either way, you need to know what you are doing (maybe by doing the mistakes, debugging them, and finally solving them). This is also why I'm saying the difference is not that big in the end.

[Siwastaja]

The main benefit of an RTOS is that each task can be written as if it owns the machine.

Really?  I think that is false confidence.  You haven't bitten yourself yet.

Spot on, except the last "haven't bitten yet" part - if you remember peter-h's posting history, all sorts of very hard to debug "my networking dies every 5 hours"  issues in projects he inherited and tried to manage have been common. It's the exact pattern of having too much non-understood complexity, and the struggle is very familiar to me: it's a HR issue, how to find capable people who know what they are doing and can manage the complexity, understand difficult things, and write good, robust code. For our company, also struggling to find the good professionals, AI was finally the solution.

[nctnico]

Right now, the biggest difference I see is in the programming style. An RTOS seems to become more useful as the system grows in size and complexity because it lets you organize the application into separate tasks, making the code easier to manage and maintain.

Not necessarily. Actually the opposite is true. In an embedded system most of the tasks are related. Seperating into more tasks means maintaining more relationships. This is a pitfall programmers tend to fall in.

Im not sure separations of concerns and implementation behind an interface call be called "Pit falls".

The "pit falls" are in doing it wrong.

If you want a lesson in how to "decouple" code corrected the litmus test is simple.  Take every single c file in your project and write a "Unit test" for it.

I don't think that will help. If you consider every C file an abstract module, these modules will need to work together in the end. In theory you could even run each module as a seperate thread with some kind of interface. So even though you can create clean, verifiable interfaces between them, this still doesn't guarantee you are going to meet time constraints.

The key to embedded software design is to identify the processes and their time constraints. How to implement these processes is the next step but coding style / modularisation are at the maintenance and verification side of the spectrum rather than at the functional side.