From Bare-Metal to Embedded Linux

[nctnico]

One disadvantage of using Linux (even RTOS!) is, is that it's never realtime!
I had this situation in a project where they moved away from using an mcu to a
system on a module (som) running Linux.
The spi bus was connected to an adc chip that generated a data ready signal every milli-second.
The new samples from the adc had to be read within 1 millisec so the data ready line was connected
to a gpio input of the som which triggered a kernel driver that read the new data and stored it in a buffer.
The os was struggling to keep up with the high interrupt rate and every now and then a sample was lost
which wasn't acceptable. In the end the solution was to put an mcu between the adc and the som in order
to create an "spi buffer" that easily read the data from the adc every millisec, buffered them and generated
a second data ready signal to the som every 32 millsec. Linux was then able to read the buffered data at a
much lower interrupt rate.
Linux does not support interrupt priorities and/or nested interrupts.

Wow, that is only 1kHz. Must have been a very slow SOM.

Several years back, I wrote a driver on the PC to handle isochronous USB data from a STM32F103 board and it had no problems with the 100kHz sample rate.

If another interrupt (disk i/o, screen, whatever) arrives just before the interrupt related to the adc chip,
than the adc interrupt routine needs to wait until the other interrupt service routine has finished.
Hardware interrupts cannot be parallelized, they cannot be spread over multiple cores.

According to my info interrupts on Linux do get spread across cores. And options like threadirqs allow to prioritise processing of IRQs preventing high priority interrupts from not being handled. As a last resort it is also possible to assign a specific core to an interrupt (in addition to assigning a specific task to a core). So for as long as you have a multi-core system, you can basically dedicate one core to a realtime task and let the other core(s) do the rest.

[HelmutDöring]

I think there is a little confusion here over what Linux is. Linux is the exact opposite of a a realtime OS. The job of the Linux kernel is not to coerce everything into proceeding in a perfectly orderly way. The job of the Linux kernel is to "boot up and get the hell out of the way", and then just keep everything else from colliding by handling interrupts at an amazing rate. Context switching and interrupt latency are not related anymore. The context switching frequency is only pertient for legacy hardware that is practically non-existant.

A lot of Linux kernel developers have done nothing more than map a hex value to a string, so that when a USB device is plugged in it can announce it's presence in the logs:

[1729871.060298] usb 1-3.4.4.4: new full-speed USB device number 103 using xhci_hcd
[1729871.152936] usb 1-3.4.4.4: New USB device found, idVendor=303a, idProduct=4001, bcdDevice= 1.00
[1729871.152944] usb 1-3.4.4.4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[1729871.152946] usb 1-3.4.4.4: Product: Espressif Device
[1729871.152947] usb 1-3.4.4.4: Manufacturer: Espressif Systems
[1729871.152949] usb 1-3.4.4.4: SerialNumber: e4b323f93cbc0000
[1729871.163725] cdc_acm 1-3.4.4.4:1.0: ttyACM0: USB ACM device

There. Faster than you can get your fingers back to the keyboard after plugging in the USB cable, a Microcontroller has been interfaced to your desktop computer. A remarkable feat considering that thousands of other things were going on at the same moment.

Slightly more adept programmers might write Serial/UART interfaces to make the relationship between the desktop and the MCU more cosy. Even more adept coders might code user interfaces that don't suck... Someday.

Linux is slightly misplaced on embedded hardware. But Linux is uniquely able to be coerced into managing vast arrays of network connected embedded systems. The CIA used a bespoke system with a Linux desktop managing an array of IBM System 360 systems-on-a-card for their world-class security-through-obscurity model that relied on the lack of real-time clocks on the slave systems to hide the order of communications!

I was a Linux kernel developer in the 1990's. The kernel today has 0 lines of the code that was present in those kernels, and is architecturally another beast altogether. In general, when you see Linux falling down at some task, you are seeing the result of a coding or architectural error, not a shortcoming in Linux. Linux is a much, much bigger thing than you think. I have built and deployed multi-thousand core Linux clusters, some of which managed thousands of VME crates. I watched the Crays, HP-X's and Intel Paragons carted out to the trash heap, replaced by Linux on commodity hardware. I haven't touched a Windows or Apple computer since 1994.

[ejeffrey]

If another interrupt (disk i/o, screen, whatever) arrives just before the interrupt related to the adc chip,
than the adc interrupt routine needs to wait until the other interrupt service routine has finished.

That's true, but it shouldn't be anything like 1 ms.  Interrupts are split into top-half and bottom half, and the top half is supposed to do only the minimum possible work to service the interrupt, and then re-enable interrupts.  There are certainly badly behaved device drivers out there, but on normal systems you shouldn't be seeing 1 ms interrupt latency.  The bottom half of the IRQ runs with interrupts enabled, only the specific interrupt being processed is blocked.  The bottom half can be interrupted by a higher priority request much like a nested interrupt on an MCU but it's implemented in software.

Anyway, if you really need bounded latency you can build with CONFIG_PREEMPT_RT which makes interrupt handlers premptable and no driver code runs in the actual interrupt context.  Although that's not always an improvement.  PREEMPT_RT is good for a complex system with lots of tasks and medium deadlines.  But the extra complexity of moving the interrupt handlers to preemptable contexts means the best case latency is worse.  So if you are willing to do a lot of tuning and you have exactly one thing you care about latency for, it can be easier to achieve that without the realtime option.

Hardware interrupts cannot be parallelized, they cannot be spread over multiple cores.

That has never been true.  As long as Linux has supported multiple processors, Linux has been able to spread interrupts across multiple CPUs and cores.  Only the local CPU has interrupts disabled during an ISR.   A very standard thing to do in linux if you need low latency in user space is to use interrupt /process affinity to put the stuff you care about on separate cores from IO handlers.

If that other interrupt takes 1 millisec, then there will be dataloss because the adc interrupt has not
been served before the new data arrives.

You shouldn't be seeing 1 ms latency on a normal system unless something is wrong.  This could be with your ADC device or something else on the system.  This can happen with devices that aren't really designed to be used on a general purpose CPU.  For instance, if there is an interrupt pin but you need to do software timed bit-banging to complete the IO.

For instance, a standard 16550A UART running at 115.2 kbps will overflow it's 16 byte RX FIFO in just over 1 ms.  But you don't really expect it to have data loss with a UART under normal circumstances, and higher speeds are common even with the old school 16550A.  For the highest data rates of course you want UARTs with bigger buffers or to not use a UART, but

[EVblog1]

Let me share how I'm currently thinking about the overall process. Please correct me if I'm on the wrong track.

Suppose I have a custom board with a processor and the required external hardware already connected, and the processor is capable of running Linux.

The first decision I would make is to use Linux as my development machine. I'd choose Ubuntu because most of the Embedded Linux tools and documentation seem to support it.

The next step would be to get a bootloader, the Linux kernel, and a root filesystem onto the board. Based on the suggestions in this thread, I'd probably use Buildroot to generate the complete Linux system. For the kernel, I could use the vendor-provided source (for example, from ST or NXP), build it, and then flash everything onto the board.

Is this a reasonable high-level way of thinking about the process, or am I missing something important?

[pcprogrammer]

Let me share how I'm currently thinking about the overall process. Please correct me if I'm on the wrong track.

Suppose I have a custom board with a processor and the required external hardware already connected, and the processor is capable of running Linux.

The first decision I would make is to use Linux as my development machine. I'd choose Ubuntu because most of the Embedded Linux tools and documentation seem to support it.

The next step would be to get a bootloader, the Linux kernel, and a root filesystem onto the board. Based on the suggestions in this thread, I'd probably use Buildroot to generate the complete Linux system. For the kernel, I could use the vendor-provided source (for example, from ST or NXP), build it, and then flash everything onto the board.

Is this a reasonable high-level way of thinking about the process, or am I missing something important?

Looks sound enough to me. Since Linux Mint is Ubuntu based it is also a very good candidate for the development system. It is what I use. The Xfce desktop is lightweight and has a nice look and feel. But that is my own preference, yours may differ.

Getting it to work in an easy as you described manner will vary based on the target you choose. I have dabbled with embedded Linux twice now and it certainly was not as straight forward. The first time was for the Allwinner F1C100s MCU and that one was not to bad, but needed the console on a different UART than the one used in the source I found. Took a bit of fiddling to get it operational.

The second time was more recent and had the Allwinner T113-S4 as target. And with this one the problem was that the full SDK for it was almost unobtainable. I had a working Linux image for a T113-S3 board I bought, but that failed on the S4, despite the fact that they are pin and most software compatible. The difference was in the DRAM for which a different initialization was needed. Took a lot of reverse engineering to get that out of the way. Another issue was getting the source to build. What I found was an old Tina Linux tree that needed all sorts of updates due to newer tool versions. The final hurdle was to get the device tree set up correctly for my own board.

It for sure wasn't smooth sailing to the finish line, but I got there. 

[PlainName]

The first decision I would make is to use Linux as my development machine. I'd choose Ubuntu because most of the Embedded Linux tools and documentation seem to support it.

What do you currently use as the development machine for your bare metal projects?

[nctnico]

Let me share how I'm currently thinking about the overall process. Please correct me if I'm on the wrong track.

Suppose I have a custom board with a processor and the required external hardware already connected, and the processor is capable of running Linux.

The first decision I would make is to use Linux as my development machine. I'd choose Ubuntu because most of the Embedded Linux tools and documentation seem to support it.

The next step would be to get a bootloader, the Linux kernel, and a root filesystem onto the board. Based on the suggestions in this thread, I'd probably use Buildroot to generate the complete Linux system. For the kernel, I could use the vendor-provided source (for example, from ST or NXP), build it, and then flash everything onto the board.

Is this a reasonable high-level way of thinking about the process, or am I missing something important?

Looks sound enough to me. Since Linux Mint is Ubuntu based it is also a very good candidate for the development system. It is what I use. The Xfce desktop is lightweight and has a nice look and feel. But that is my own preference, yours may differ.

Getting it to work in an easy as you described manner will vary based on the target you choose. I have dabbled with embedded Linux twice now and it certainly was not as straight forward. The first time was for the Allwinner F1C100s MCU and that one was not to bad, but needed the console on a different UART than the one used in the source I found. Took a bit of fiddling to get it operational.

The second time was more recent and had the Allwinner T113-S4 as target. And with this one the problem was that the full SDK for it was almost unobtainable. I had a working Linux image for a T113-S3 board I bought, but that failed on the S4, despite the fact that they are pin and most software compatible. The difference was in the DRAM for which a different initialization was needed. Took a lot of reverse engineering to get that out of the way. Another issue was getting the source to build. What I found was an old Tina Linux tree that needed all sorts of updates due to newer tool versions. The final hurdle was to get the device tree set up correctly for my own board.

It for sure wasn't smooth sailing to the finish line, but I got there. 

Using an obscure Chinese CPU with next to no documentation is not helping. I can highly recommend using a board with an iMX SOC from NXP. These are also targetted at low volume products so support and documentation are easy to get. There are many boards out there. At this moment the iMX8 is a good choice (mature but not too old). Also, NXP has a forum with lots of information and several knowledgeable NXP employees are actively participating.

[pcprogrammer]

Using an obscure Chinese CPU with next to no documentation is not helping.

Tell me about it 

For me the lure was the price and the fact that it is eLQFP. I bought 10 of these T113-S4 MCU's for 67.51 euro on, you guessed it, Aliexpress. The price has doubled since. I have not looked at NXP MCU's, but the ones on the Teensy boards are BGA, and that is a bridge to far for me.

It was also kind of the point of my post. Using a well documented and supported MCU or board makes life a lot easier.

[spostma]

I am not speaking from experience, but I think that dietpi is the first distribution to consider for embedded linux projects.
https://dietpi.com/

They describe their project as an "highly optimised minimal Debian OS".
they support every embedded board you can think of, and have an active forum.

[EVblog1]

What do you currently use as the development machine for your bare metal projects?

At the moment, my main development machine is a Windows 11 laptop, and all of my bare-metal embedded projects have been done on Windows.

To start learning Embedded Linux, I installed Ubuntu in a virtual machine. I also had MySQL installed for one of my projects, but it made my laptop noticeably slower, so I removed it. For now, I think using a virtual machine is the most practical approach. If I find that I'm spending most of my time doing Linux development, I'll consider switching my laptop to Linux.

Based on the suggestions in this thread, I'm also think I'm going to get an NXP-based development board. In the meantime, my plan is to set up the required development tools, get familiar with the workflow, and practice building the Linux kernel. That way, when I finally have the board, I can spend more time experimenting rather than setting everything up from scratch.

I know it won't be as simple as just flashing a board and having everything work, but doing something is better than doing nothing. The important thing is to keep learning and keep moving forward.

Cheers!

[langwadt]

What do you currently use as the development machine for your bare metal projects?

At the moment, my main development machine is a Windows 11 laptop, and all of my bare-metal embedded projects have been done on Windows.

To start learning Embedded Linux, I installed Ubuntu in a virtual machine. I also had MySQL installed for one of my projects, but it made my laptop noticeably slower, so I removed it. For now, I think using a virtual machine is the most practical approach. If I find that I'm spending most of my time doing Linux development, I'll consider switching my laptop to Linux.

on win10/11 just install WSL

[PlainName]

What do you currently use as the development machine for your bare metal projects?

At the moment, my main development machine is a Windows 11 laptop, and all of my bare-metal embedded projects have been done on Windows.

To start learning Embedded Linux, I installed Ubuntu in a virtual machine.

[...]

 The important thing is to keep learning and keep moving forward.

OK, it's swings and roundabouts. Using Linux would be a good idea because your project will be using Linux, so you have a better understanding of what's going on.

The downside is that in the VM you lose access to all the Windows stuff you've been using (if, indeed, you have been using stuff!) like clipboard utills, file managers, etc. And you start to fall into the trap of needing to do development on whatever hardware you are using. So if you have a Pi you'd be using a Pi for development, as an example. Lots of people do that and don't think it's strange although they have a PC 10 times bigger and faster doing nothing.

If you stick with Windows you don't get any help from the development system, but what you do learn is cross compiling (which used to be the norm, once upon a time). You then have the knowledge and skills to develop any product on any platform so long as there is an appropriate cross compiler. Linux then becomes just another project, albeit rather large. You find out exactly what you need to build, and how, to get the thing working whereas with OS support that can be somewhat hidden

At your stage that may be a step too far and you need to walk before you can run, so learning all about Linux on Linux would be a good start. But keep in mind that there are other perfectly valid ways of achieving the same end.

[EVblog1]

Is anyone here run Linux on the STM development board?

[pcprogrammer]

Is anyone here run Linux on the STM development board?

Which one? There are many boards with STM processors on them, but most are not suited for running Linux.

If you want something less expensive than a Teensy board, you could look into Milk-V boards. https://milkv.io/duo

I have not build Linux for them, but have ran an image on one of these to test the board. There is plenty of information around for these boards.

Or you could look into one of the LuckFox boards. https://wiki.luckfox.com/Luckfox-Pico-Ultra/

[EVblog1]

Which one? There are many boards with STM processors on them, but most are not suited for running Linux.

I initially thought I could use my STM32F407 Discovery board to learn Embedded Linux, but I found out that it isn't capable of running Linux, so I'll need different hardware.

Out of curiosity, which board(s) do you mostly use for Embedded Linux development?

[uer166]

I don't develop Linux but whatever you do, I heard that you should use something that has upstream/mainline Kernel support. This means you're not beholden to the board/MPU vendor to provide you the kernel sources and use their development tools/environment (Yocto/Buildroot etc).

Examples:
STM32MP1 does *not* have mainline Linux support so you're beholden to ST and their old Kernel sources.
NXP imx8m and friends *do* have it so it's supposedly a lot smoother.


The classic board that people learn this on is the Raspberry Pi, however there's no complete open Broadcom MPU documentation so I actually have no clue how they do it.

[Karel]

I too recommend using a Raspberry Pi for starting & learning Linux, it has the best support (community, documentation, software).
Once your project runs fine on the Raspberry Pi, you can port your code to some other board if you wish.

[pcprogrammer]

Which one? There are many boards with STM processors on them, but most are not suited for running Linux.

I initially thought I could use my STM32F407 Discovery board to learn Embedded Linux, but I found out that it isn't capable of running Linux, so I'll need different hardware.

Out of curiosity, which board(s) do you mostly use for Embedded Linux development?

A board with a STM32H7 variant might be able to run Linux, but I have not looked into that, despite having such boards. Yes, I'm a MCU dev board hoarder 

I to, second the Raspberry PI as a starting point. I have used the PI2B for a central heating system that has run for several years, but now has been replaced with a heat pump system.

[EVblog1]

Okay, since several people have recommended starting with a Raspberry Pi, I'll use that as my learning platform. I already have a Raspberry Pi 3B+ and have now installed fresh Raspberry Pi OS on it. It took me a little while to get everything working because I'm running it in headless mode, but it's up and running now. Hopefully I can start experimenting with Embedded Linux on it.

[EVblog1]

Now that I have the Raspberry Pi set up, I know there are plenty of tutorials showing how to write a simple "Hello World" or LED blinking kernel module. Those are useful for learning the mechanics, but I'm looking for a more meaningful problem statement that actually justifies writing a kernel module or a device driver.

In other words, I'd like to think with a real problem where the correct solution is kernel or driver development, rather than forcing an LED blinking example into the kernel just for the sake of learning. I'd like to understand not only how to write one, but also why it belongs in the kernel instead of being implemented as a user-space application.

Have any of you used the Raspberry Pi for a meaningful Linux kernel or device driver project? If so, what was the problem you were solving, and why did it require kernel or driver development instead of a normal user-space application?

[PlainName]

Set of buttons, arranged in a cross, you can press to.. do stuff. Could write it as an actual keyboard input but using GPIO rather than USB to make it simple in hardware and software, and existing modules would still be useful to get ideas:

https://kernel.org/doc/html/v4.12/input/index.html

[Kilrah]

In other words, I'd like to think with a real problem where the correct solution is kernel or driver development

Well the whole point of using linux is usually to avoid having to do any of that, so normally you would never need to touch the kernel code, at most you'd maybe recompile it with a different set of options based on what you need or don't.
You only would e.g. if there is no driver for some of your SoC's peripherals or the existing one is bad/lacks functionality (but then you need documentation to write the driver, and usually if the driver's bad you'll have a hard time getting the doc too...), or you develop a custom device you connect via PCIe/USB/... that you cannot interact with using any of the widely established standards and thus have to develop a custom driver for.

You could use e.g. a Zynq board, write some custom logic like a weird UART with specific functionality and different registers from a standard 16650, then write the driver for that so you can inteact with it from userspace either through the standard /dev/tty or a completely different thing you develop a protocol for and write the corresponding driver,...
But in general the whole point is that there are standards for pretty much everything, linux can handle basically all of them, and you use it precisely so you don't need to roll your own anything low level and can just concentrate on your application code.

Keyboard example is good.

[SiliconWizard]

According to my info interrupts on Linux do get spread across cores. And options like threadirqs allow to prioritise processing of IRQs preventing high priority interrupts from not being handled. As a last resort it is also possible to assign a specific core to an interrupt (in addition to assigning a specific task to a core). So for as long as you have a multi-core system, you can basically dedicate one core to a realtime task and let the other core(s) do the rest.

threadirqs makes the kernel handle IRQs as kernel threads, so you benefit from the scheduler (priority and all preempting niceties). Without threadirqs, IRQs are handled just via a single interrupt handler (AFAIK), giving you potentially lower interrupt latency (for a single given IRQ) but making it unable for other IRQs to preempt one currently being handled (if I'm not mistaken, please correct me if I'm wrong), so that in a typical system where there's a lot of interrupts coming in, the average latency is actually worse.

While with threadirqs, IRQs can get scheduled on different cores just like any other kernel thread, I'm not completely sure this is the case without threadirqs, but if someone knows the kernel at this low level, please chime in.

Edit: after digging a bit, without threadirqs, however the IRQs will be handled is almost entirely dependent on the underlying system. On systems which have an interrupt controller with preemptible interrupts, one IRQ can preempt another one if it has higher priority, but it depends on the interrupt controller entirely, and the priorities may or may not be configurable (as I remember, older x86 systems had fixed interrupt proprities). A not too old x86 system can probably dispatch interrupts over all available cores, but I admit I haven't looked in details how the interrupt controllers work on recent x86 systems.

In any case, threadirqs allows more fine-tuning and allows other kernel tasks to preempt IRQ handlers, which I don't think is possible without threadirqs. So, depending on the underlying interrupt controller and your exact use case, one may be more adequate than the other.

[Tation]

In other words, I'd like to think with a real problem where the correct solution is kernel or driver development, rather than forcing an LED blinking example into the kernel just for the sake of learning. I'd like to understand not only how to write one, but also why it belongs in the kernel instead of being implemented as a user-space application.

Only did that inside FPGAs, where both the (in-house) peripheral and the CPU shared the same silicon. Today I will try to connect the peripheral thru any capable enough (for such peripheral) standard interface already managed by Linux. Less effort and less headaches.

[ejeffrey]

threadirqs makes the kernel handle IRQs as kernel threads, so you benefit from the scheduler (priority and all preempting niceties). Without threadirqs, IRQs are handled just via a single interrupt handler (AFAIK), giving you potentially lower interrupt latency (for a single given IRQ) but making it unable for other IRQs to preempt one currently being handled (if I'm not mistaken, please correct me if I'm wrong), so that in a typical system where there's a lot of interrupts coming in, the average latency is actually worse.

While with threadirqs, IRQs can get scheduled on different cores just like any other kernel thread, I'm not completely sure this is the case without threadirqs, but if someone knows the kernel at this low level, please chime in.

Without thread IRQs, IRQs are handled on the CPU where they are delivered.   However, platform interrupt controller such as the x86 IO-APIC and Local APIC have the ability to direct interrupts to specific CPUs.  This was the original use case of MSI-X: traditional MSI uses a single memory address for all IRQs which on older systems delivered them all globally.  With MSI-X, each interrupt can have a different memory address.  This allows the OS to distribute interrupts to specific cores.  Modern processors have interrupt remapping which can deliver interrupts to specific CPUs just as well for traditional MSI.

Also, even without threaded IRQs, it's is normal for IRQs to re-enable interrupts partway through their execution.  This is the "top/bottom half".  In this case, the interrupt handler cannot be preempted by user space, but it can be preempted by another interrupt.

You are correct that threaded IRQs will generally increase median latency.  The goal is to reduce worst-case latency by avoiding being held up by slow running interrupts.  However, plenty of real-time Linux systems do not actually use this.  If you only have one specific task that is "real time" you can just dedicate one or more cores to this.  You can send all other IRQs to the houskeeping cores.  This is often the way to get the absolute lowest latency.  Threaded IRQs is useful on a more complex system where you have too many tasks to pin them to specific cores.

I have measured this with a low latency NIC.  It's possible to get < 2 microsecond round trip latency for minimum size UDP packets on an off the shelf x86_64 system just by using a fairly simple set of tools.