From Bare-Metal to Embedded Linux

[EVblog1]

There are questions about the OP's actual level in coding.

I generally write C programs for microcontrollers. I have tried multithreading with FreeRTOS while learning it, but I haven't really needed C++ in my project, so I haven't used it much. I do have a basic understanding of OOP concepts, though.

My approach is to understand the problem and find a solution before jumping into coding. The problem is that right now I don't really have a practical problem to solve. I want to develop a device driver that the Raspberry Pi doesn't already have and then test it on my Pi. I have some SPI and I2C devices and a few sensors that I can use for experimentation.

So far, I have also experimented with controlling an LED from a kernel module and through a character driver. But I don't want to just write code for the sake of writing code. I'd rather work on something that has a real purpose and forces me to understand the Linux driver-development process properly.

[nctnico]

Typing it back to the point on bloat of boost and the likes.  Your belief that the lower levels are "simpler" however is correct and false.  Usually when you start with simpler lower level blocks the intent is to assemble them into something greater.  As pointed out, when you let everyone in a code base do that you end up with more than 1*n versions.  Each developer will use multiple different ways, constructs and patterns in their own work, let alone everyone in the team.  Giving the team larger, pre-fab, blocks makes the wall not look like a child built it.  Where the code YOU wrote on Tuesday doesn't work well with teh code you wrote on Thursday.

Yep. The discussion got hung up on Boost specifics but your reply touches the core precisely. One of the competences you need as a software developer is to determine when to use libraries and when not. If a library is desired, the next question is which library is the most appropriate to use.

Many libraries are written from the idea 'hey, I can make access to this simpler'. Sometimes that is a succes, sometimes not (too buggy, too complicated). This also depends on what is there already. Using the pthread library as-is is not that simple and with C++ things like mutexes which release when they go out of scope (lock_guard) are super handy (fire and forget) so having a layer on top of pthreads is very helpfull. OTOH, the BSD style socket API is pretty good as-is and every mainstream OS supports it.

[tellurium]

OTOH, the BSD style socket API is pretty good as-is and every mainstream OS supports it.

I'd argue BSD socket API is good, but that's an offtopic. I've made a video on BSD sockets API where I elaborate on that

[paulca]

There are questions about the OP's actual level in coding.

I generally write C programs for microcontrollers. I have tried multithreading with FreeRTOS while learning it, but I haven't really needed C++ in my project, so I haven't used it much. I do have a basic understanding of OOP concepts, though.

My approach is to understand the problem and find a solution before jumping into coding. The problem is that right now I don't really have a practical problem to solve. I want to develop a device driver that the Raspberry Pi doesn't already have and then test it on my Pi. I have some SPI and I2C devices and a few sensors that I can use for experimentation.

So far, I have also experimented with controlling an LED from a kernel module and through a character driver. But I don't want to just write code for the sake of writing code. I'd rather work on something that has a real purpose and forces me to understand the Linux driver-development process properly.

I get the chicken and the egg thing on personal learning projects.  They need a valid "driver" and ideally a "Product owner" role.  Providing that all yourself is a bit of a faff.  You get to over-rule your own requirements anytime it gets painful... maybe thats the good part.

Multi-threading / Multi-processing on "big iron" is NOT the same thing as on an MCU.  An MCU is typically a mono-processing entity.  Yes concurrency happens in it's environment and yes it's peripherals are technically concurrent to it's processor, however in 99% of cases it has a "single core".  Also as it's most like "ARM32" it's a linear pipeline.  It is therefore almost entirely "deterministic".

Now consider on big iron.  Memory is async and has read/write cycles long enough to execute a large pipeline of speculative instructions BEFORE the reply comes back from memory to determine which way the "Bra" statement choose.  In some cases they can execute BOTH pathways speculatively and then roll out or purge the incorrect one later.

You have multiple cores and probably multiple CPUs.  When you "fork" a line of code 64 times, it will schedule 64 processes and a "big iron" box will handle all of them, probably concurrently on different cores, like "simultaneously" in microseconds.

Then you have not one cache, but 3, 4 or 5 layers of it between the CPU and the RAM.  When once core moves a sentinel in memory.
If the other cores walking that structure are looking for the sentinel in the cache, but "which cache"?  Each core has it's own cache.  Groups of cores have a shared cache, the CPU has many of those wrapped in a further cache.  The RAM controller has  cache.  When you update your sentinel chances are the threads already in a pipeline will complete execution and write back, at which point the CPU pipelining system "should" evict it for a cache collision.  Flush the relevant caches, abort all inflight pipeline chunks and start again.  "Pipeline stall".

It should be noted that in "very high performance" systems, where "latency must be deterministic and with SLAs",  threads are not stypicaly the solution.  Not in the critical path anyway.  Simple because they are "non-deterministic".  In the "next gen" stock exchange gateways, the "2 threads per session, one in, one out" was dropped in favour a single thread processing across multiple read and write sessions.  One thread per core.  Static assigned affinity. 128 cores.  Each broker session was locked to a core and the two "threads" simple roundrobined in a hot loop.  "Can read?", "Can write?".  These threads could even span multiple brokers and round robin around many producer/consumers.  The result was a notable increase in performance due to less "contention, context switching and pipeline stalls".  The failure mode explained to me made it sound simple.  Every single one of those threads is immediately coupled to a read or write from the NIC DMA controller, eventually.  A single NIC can only send one ethernet frame at a time.  Yes there can be bonded links, but even then it's not quite that simple.  They all hit the same monitor zone in the OS and got stalled and suspended on the same signal.  There really was no concurrency at the "effects" stage.  Processing the NIC queues sequentially with a single thread solves that quickly.  Now they are processed in an order the hardware can immediately move with.

Each physical CPU when there are many, have their own caches.

So you move from a word where in "Arm32" the vast majority of instrucitons including memory reads and writes are "single clock atomic".  You have "zero" or limited "data cache" and only instruction caches.  Minimal pipelining.  Single core.  etc. etc. etc.  A MUCH simpler world.

However.  In a way that makes it WORSE, not better.  The concurrency issues or missing monitor zones may not become as visible on such a system, but they still exist.  This leaves you into a false sense of security because it all appears to work... right up until it doesn't.   An unexpected ordering of interrupts.  A hung interrupts stalls timers and causes out of ordering in your processing.  Then suddenly things stop working and stop in a very hard to work out why way.  "In-deterministic" is usually the result in concurrent bugs.  The output is effectively garbage with no "Sender address".

[EVblog1]

So I’m trying to understand what exactly Device Tree is. https://www.devicetree.org/

Basically, somewhere in the code, we need to tell Linux kernel about the hardware for example, which pin it is connected to, what the I2C device address is, which interrupt it uses, etc.

We can configure these things in the application code or in the driver code, right? A driver already has its own files where we can configure this hardware information.

But I’m not really understanding the advantage of using Device Tree here. What problem does it solve, and what benefit do we get by keeping this hardware information separately in the Device Tree instead of putting it directly in the driver or in application code?

[Kilrah]

A driver already has its own files where we can configure this hardware information.

But then you have to modify driver files and recompile the kernel for that specific piece of hardware.
DT allows more "dynamic" definition of where the hardware lives without having to do that. DT says "there's an i2C port at this address with these capabilites", kernel can then go "great, i'll tell the generic driver that's what to use" and your application just sees the i2c port. So neither the kernel nor your app needs to be tailored to the specific hardware, just the DT file.

Once again your approach of "I want an example of having to write a kernel driver / modify kernel code" is backwards, a lot of effort is made to avoid ever having to do that. If you need to do it it's because all the solutions that are used to avoid it have failed, i.e. your system really has special needs because the majority cases are well covered.

[pqass]

But I’m not really understanding the advantage of using Device Tree here. What problem does it solve, and what benefit do we get by keeping this hardware information separately in the Device Tree instead of putting it directly in the driver or in application code?

Linux runs on more computers/boards than just x86 with classic i/o.
For example, rather than having 0x3f8 hardcoded in the driver, the base address of the serial port is given to it instead (at runtime).
The same 16550 UART code is compiled into the kernel just once but the hardware can be anywhere in the address space.
Why copy the same driver code under a different name with just the hardcoded base address (or select few, incl. 0x2f8) changed?

"The purpose is to move a significant part of the hardware description out of the kernel binary, and into the compiled device tree blob, which is handed to the kernel by the boot loader, replacing a range of board-specific C source files and compile-time options in the kernel." 
From: https://en.wikipedia.org/wiki/Devicetree#Linux

[EVblog1]

Is it necessary to understand the root filesystem when writing a general application, like controlling an LED from a switch? Or is understanding the root filesystem mainly necessary when we are creating a custom Linux image?

I’m asking because I’m currently trying to understand things at a higher level, rather than going into the details of the entire filesystem.

[paulca]

Is it necessary to understand the root filesystem when writing a general application, like controlling an LED from a switch? Or is understanding the root filesystem mainly necessary when we are creating a custom Linux image?

I’m asking because I’m currently trying to understand things at a higher level, rather than going into the details of the entire filesystem.

Two paths.  Depends where YOU draw YOUR line.

If you want to remain in the "Unix / POSIX" world and comply with that eco-system and standards, then you MUST use the SFS layout and use it as your interface.

"Everything is a file".  That is the Unix way.  That is the Unix programmer interface.  You "can" sys call the kernel and ask about the CPU, but it would prefer your open a file handle on /proc/cpu hierarchy and look yourself.  Thus the ACL file perms can manage it seemlessly.

Code: [Select]

cat /proc/cpuinfo

Just look at how machine and human parsable that is.  Hint.

In your case, I see a /dev/my_led at least.  Possibly a /dev/my_switch.  Your app opens file handles to both and does what it does in user space.  Your dev files are assigned to your kernel driver/drivers via "udev" or systemd.

If you do (DONT!) as root:

echo 1 > /dev/sda

It will literally.... and I mean literally, write a 1 into the first addressable byte of that Hard disk.  EDIT (and a newline!)

cat /dev/sda

WILL dump it contents to standard out.  Or your terminal, which will not be wise.

To set the speed of my CPU fan to 50% I can do something like:

echo 128 > /proc/sys/bus/pcie/0110/pwn_5

The other way...

Draw the line at "Thanks Linux, I'll take over now".  Just do whatever you want.  Run your app as root, let it peek and poke directly into /proc/sys/bus/pci or make direct syscalls into the kernel for that same access,  whatever floats your boat.  At the "per package" "source code" level, Linux is more like an OS building toolkit.  Your way.

The cost is compatibility.  If you need to move to another kernel or anoher POSIX OS, you would need to rewrite it.

EDIT: One area you can very likely ignore completely is the entire "multi-user" aspect.  When setting things you avoid being dragged into multi-user requirements. 

Linux broadly speaking boots to one of a few specific levels:
0 - Bare root console - physical access hack via your EUFI bootloader.
1 - "Single user" - the above but an actual shell and basic init, like auto mounts.  No "login getties", no networking.
2 - "Multi-user" - often just combined with the next.
3 - "MU + Networking" 
4 - never seen it used.
5 - Full GUI boot with GUI based login management.
6 - reboot
Im not even sure "init" accepts "init 0" which might be shutdown if it exists at all, but I put "0" level as a hard "init=/bin/bash" boot level.

In times gone by you used to be able to just "init 1" and get a root terminal with everything shutdown and just "init 5" it to bring it back up.  These days however, multi-user linux distros block access to the init 1 terminal and as root hasno password set in /etc/shadow it fails.

[netmonk]

Are you going to bill a customer for every task running on the CPU?

If not, then perhaps you don't need Linux.

Unix emerged from an era in which expensive computing resources had to be shared between multiple users, departments and workloads. Processes, schedulers, priorities, accounting and time-sharing all make perfect sense in that context.

But this historical model has become so deeply embedded in our thinking that we now tend to treat it as the natural way a computer must work.

It isn't.

A dedicated embedded system may have one owner, one purpose, a known hardware configuration and a largely predictable workload. In that context, importing the complete Unix process/scheduler/resource-accounting model can amount to importing a datacenter abstraction into a toaster.

Of course Linux can be made sufficiently deterministic for many real-time applications. PREEMPT_RT and decades of engineering prove that.

But notice what we are doing: we start with a general-purpose system built around concurrent processes and dynamic resource arbitration, and then spend considerable engineering effort constraining its nondeterminism.

There is another possible approach:

start from determinism.

If you want to learn Linux, absolutely run it on your desktop, dual-boot it, put it in a VM, break it, rebuild it, understand it.

But let linux embeded dies and rot in peace, in what i would call the worst use case of linux. 

[abeyer]

Are you going to bill a customer for every task running on the CPU?

Everyone knows that subscription pricing is the future... why not try? 

Unix emerged from an era in which expensive computing resources had to be shared between multiple users, departments and workloads. Processes, schedulers, priorities, accounting and time-sharing all make perfect sense in that context.

emphasis added to point out why this may still make sense... Multiple workloads in an embedded context is perfectly reasonable and common, even when it's effectively a single "user", and can have similar needs for scheduling, prioritization, accounting, time-sharing, and privilege separation. The unix model isn't the only way (nor even the best) to achieve this, but it's a well known and understood approach.

Of course Linux can be made sufficiently deterministic for many real-time applications. PREEMPT_RT and decades of engineering prove that.

But notice what we are doing: we start with a general-purpose system built around concurrent processes and dynamic resource arbitration, and then spend considerable engineering effort constraining its nondeterminism.

There is another possible approach:

start from determinism.

You act like hard determinism and realtime are part and parcel of "embedded", when there are lots of use cases for embedded devices where they may not be necessary or even desirable.

If your point is "Don't use linux for hard realtime", then sure, that's a reasonable default. But you seem to be making much more sweeping generalizations.

[SiliconWizard]

Just like with programming languages, libraries, "frameworks"... Linux is often chosen for a given "embedded" target either because it's the only readily available solution (many SoCs do not officially support anything else, and many are not publicly documented well enough to reasonably embark on that journey on your own unless you have unlimited time), or because it offers "immediate" and proven support for many peripherals, filesystems, network stacks, video... Getting all that without an OS such as Linux is hard work. Those are by far the main reasons for choosing Linux on embedded, not its specific scheduling or memory handling merits.

I'm all for lightweight and more adequate alternatives, but most often, if Linux is considered for a given target/application, there just isn't any reasonable alternative. So we can debate until the cows come home. I have implemented baremetal support for some such SoCs (for instance CV1800B and Spacemit K1), but it's definitely no picnic. And on the K1, I haven't even touched the GPU part. Others have done that for the RPi's, and there is the Circle library (C++), and Ultibo (Pascal), although for the latter, I don't know where the project is going. It seems on hold since 2022 or so.

If you *need* to use a SoC running Linux (for reasons mentioned above), then the most reasonable approach is to use a secondary MCU/core dedicated to low-level, time-critical tasks.

And if you need a powerful SoC (for its raw performance) but do not need/want Linux, good luck because you're usually entirely on your own. Yes, we could wish vendors would release a lot of doc publicly and some reasonable baremetal support files, but it usually doesn't happen, and for that to happen, there would need to be huge demand, and there just isn't.

[nctnico]

I always find it interesting when people talk about realtime without specifying numbers. Even on a desktop any OS handles lots of realtime processes. Like playing audio & video, dealing with devices that have strict interrupt latencies (USB for example). And the whole non-deterministic stance against Linux is flawed. The entire definition of an embedded system is that it has very well defined tasks. So such a system is deterministic in nature. It is not like a user is going to play a game of Doom. Or load a spreadsheet. Where it comes to embedded Linux (and any embedded system) it is all about software architecture, resource planning and good design choices.

[netmonk]

I agree that "realtime" without numbers is mostly meaningless.

A 20 ms deadline, a 200 µs deadline and a 2 µs deadline are three completely different engineering problems.

And yes, a carefully engineered Linux system can absolutely satisfy many such constraints.

But this is slightly different from what I mean by determinism.

When you say that an embedded system has a well-defined set of tasks and therefore is deterministic "in nature", I think this is exactly where the interesting distinction lies.

The application may be deterministic in intent, while the execution substrate underneath it remains dynamically scheduled and only statistically or boundedly predictable.

That can be perfectly acceptable. Often it is.

My question is whether, when the workload is already known and bounded, we necessarily need to introduce a general-purpose dynamic scheduling model in the first place.

In other words, instead of asking:

"Can Linux meet my deadlines?"

I am asking:

"How much runtime arbitration do I actually need if the machine's behaviour is already known at design time?"

That is not really a Linux criticism.

It is a question about software architecture.

(also we all noted that yourself is subject to your first criticism)

[PlainName]

The entire definition of an embedded system is that it has very well defined tasks. So such a system is deterministic in nature. It is not like a user is going to play a game of Doom.

https://youtu.be/9vad2zkV5Ro

[SiliconWizard]

Actually, video games may not be the best counter-example, because they often are pretty much designed with similar approaches to embedded development: "real-time" constraints, entirely custom memory allocators (such as bump, pool allocators and the like), and if using multiple threads, well-defined tasks distributed over specific threads.

[nctnico]

Actually, video games may not be the best counter-example, because they often are pretty much designed with similar approaches to embedded development: "real-time" constraints, entirely custom memory allocators (such as bump, pool allocators and the like), and if using multiple threads, well-defined tasks distributed over specific threads.

Exactly. And most mainstream OSs support these kind of applications for this reason alone. You can hang up entire execution chains across multiple processes based on the vertical retrace interrupt. A method which dates back to the first home computers. Every thread / process is held until a new interrupt fires which then causes a flurry of activity like reading new data from disk, DMA transfers, GPU rendering, etc. Each with a maximum alloted time to execute in order to make it in time for the next vertical retrace.

[NorthGuy]

Actually, video games may not be the best counter-example ...

A good example of a very hard-time (in ps) thing that usually runs Linux is an oscilloscope. But the real-time part is done by FPGA and ADC, not by the chip running Linux.

Same on PC. The OS(and CPU) doesn't really do anything real-time - peripheral devices do.

[paulca]

I've been in <100uS SLA land and produced <1uS consistent wire to wire (nic to nic) latency on a "GP OS".  It's not impossible.  Sub micro-second timing.

Ironically the part those figures dont include is the hardware time.

[Siwastaja]

Video games and multimedia are similar to real-time embedded projects - and are not.

Both are designed with similar real-time concepts, sure. There is a lot of knowledge to learn and share the both ways. But the important small difference is the reliability factor: in video games and multimedia, missing a deadline is a real nuisance which developers want to avoid and minimize, but occasional miss is still entirely OK. If you are designing a space rocket or pacemaker, you just guarantee 100% that no deadlines are ever missed. 99.9% vs. 100.0% guarantee sounds like a small difference, but it has significant consequences: the solutions look very different: although both Linux and Windows NT kernel can satisfy the timing needs of serious gamers, they are never used in any real-time system where missing a deadline has a cost, like death (of a human being, or even just a transistor) - if someone tries, they stop trying after 5 blown transistors during the first hour at lab with the prototype.

There are hobby projects that tried nevertheless, linuxCNC with the old parallel port mode as a good example. These guys went to great lengths, creating their own linux distribution, fine-tuning it, creating a timing analysis tool which looks for worst-case numbers while you are supposed to browse web and vigorously move windows all around - and yet it didn't work but had a tendency of creating slightly wobbly routing/cutting every now and then, and everyone transitioned away from it and moved the real-time stuff to some small AVR microcontroller which takes gcode through UART from the same linuxCNC UI.

[paulca]

The 4 9s it's called.  99.99% uptime.

Which sounds great until you realise that accommodates an hour outage once a year.

[PlainName]

missing a deadline is a real nuisance which developers want to avoid and minimize, but occasional miss is still entirely OK. If you are designing a space rocket or pacemaker, you just guarantee 100% that no deadlines are ever missed.

There is quite a range between those two extremes, and it's very close to the space rocket end before a missed slot is actually fatal (to the product).

linuxCNC with the old parallel port mode as a good example. These guys went to great lengths, creating their own linux distribution, fine-tuning it, creating a timing analysis tool which looks for worst-case numbers while you are supposed to browse web and vigorously move windows all around - and yet it didn't work but had a tendency of creating slightly wobbly routing/cutting every now and then, and everyone transitioned away from it and moved the real-time stuff to some small AVR microcontroller which takes gcode through UART from the same linuxCNC UI

Contrast with Mach3, which ran on Windows very well indeed. Preferred an OS without the bloat, so W2K or XP, but there is no external hardware or controllers involved. Never had a stutter or wobbly from it yet.

Having said that, it suffered the usual problem of the developer figuring a rewrite would make it twice as good and then finding out it's a just a way of perpetually not getting anywhere. I think he did manage something in the end, but external boxes to deal with jitter are now de rigueur.

[Siwastaja]

missing a deadline is a real nuisance which developers want to avoid and minimize, but occasional miss is still entirely OK. If you are designing a space rocket or pacemaker, you just guarantee 100% that no deadlines are ever missed.

There is quite a range between those two extremes, and it's very close to the space rocket end before a missed slot is actually fatal (to the product).

Yes, and there is this huge segment of cheap commodity items that still need hard realtime, or they fail and cause material damage. Like, a $200 variable frequency drive which has code to protect its IGBTs from blowing up. No one would even think about using Windows or linux kernel to do that low-level control. It's MCU, FPGA, CPLD or combination thereof. For higher-end UX functionality, there might be separate linux/Windows computer which handles the non-timing critical functionality.

[PlainName]

I think we are still on extremes, here. It can be rather more murky even given onerous specs.

For instance, the spec may state that some pulse needs counting and every pulse is unmissable, and they are very high frequency. So you look at what an interrupt could handle and it might be right on the brink of guaranteeing every trigger is caught. Just needs a momentary lapse and you're stuffed. But maybe that chip has a counter which can easily count all pulses, and you only need to read the counter before 32,535 of them have been seen. Much, much less onerous yet still keeps to the real-time specs.

Of course, much depends on what you do with the data, and maybe you then have to react within a certain time or can be a bit more flexible. So that's why I say a blanket "<popular OS> isn't suitable because it's not real-time" is just flopping to one extreme with considering what might actually be appropriate for a given project.

[Siwastaja]

I think we are still on extremes, here. It can be rather more murky even given onerous specs.

We don't need to be on extremes; extremes may be easier to understand though. But a motor controller FOC is a perfect example of "everyday" commodity stuff (millions produced, cost in dollars, see e.g. every e-scoot e-waste product) which fails in smoke in minutes if you try to run it on linux. No one just ever does it, because they know it can't be done. They know linux isn't a realtime system, basic knowledge you can read everywhere.

It is just so obvious to you guys not to run it on linux/Windows you don't even consider the idea. But that's exactly the core of the "linux isn't a realtime OS" argument. By saying "it can do multimedia", oh yes, it can, but multimedia is not a realtime embedded control application like driving a $10 e-scooter motor is. This is just a nctnico's recurring brainfart. He's wrong. Multimedia isn't the same. Screen stutter is entirely normal and relatively harmless. Case closed.

For real-time, it's bare metal or RTOS. For something which can stall for tens of milliseconds, in rarer cases for seconds, for unexpected reason, linux is fine. linux can be tuned to perform better, but at some point you are using a lot of effort (the historical linuxCNC case) on a battle you can never win. It makes no sense when the $1 microcontroller does the trick and can communicate with the linux computer through UART, so you get the benefit of both. The hardware on the linux box has a FIFO on it's UART exactly because they know linux can't do even the UART real-time. This solves the problem. Higher-level comms, hardware to deal with the timing uncertainty, partitioning the responsibilities to where they belong.

The benefit you are getting from Linux (or, say, Windows CE if you believe in that) is exactly that you get a large stack of usable stuff that doesn't need huge amount of tuning.