From Bare-Metal to Embedded Linux

[EVblog1]

My background is in bare-metal embedded development. In that environment, the software architecture is fairly straightforward: I write reusable peripheral drivers (UART, SPI, I2C, GPIO, etc.), and then I build the application logic on top of those drivers.

What I'm trying to understand is how this changes in Embedded Linux.

I understand that Linux already provides drivers for many peripherals and devices. My confusion is about when and why a developer writes kernel code at all

For example, what kinds of problems are typically solved in kernel space rather than user space? Is kernel programming mainly about writing device drivers for new hardware, or are there other kinds of functionality that belong in the kernel? What criteria determine that some logic should live inside the kernel instead of as a normal application?

I'm not asking how to write a driver yet. I'm trying to understand the software architecture and the role of kernel programming in an Embedded Linux system, so I can build the right mental model before diving into the code.

To get involved, I've already installed Ubuntu Linux in a virtual machine and started exploring the Linux environment. I'm also planning to buy an embedded board that supports Linux so I can get hands-on experience with kernel and driver development. My goal is to understand the concepts first, then apply them on real hardware rather than just following tutorials blindly.

I'm not sure how many people here work with Embedded Linux, but I thought this would be a good place to ask

Thanks again for taking the time to help.

[shabaz]

Not sure you'll find many neatly-packaged tutorials regarding writing your own device drivers, although I've not checked recently. You may even need to delve into FPGA/SoC tutorials to pick up bits of information here and there, that you can also apply elsewhere. However, from your question, I think you need to first broaden your knowledge, if you're not aware what issues could be occur, that may necessitate such a driver.

It is worthwhile first understanding Linux in general, and then try to write something real-time, and see what happens when (say) trying to control an output or read an input at specific rates. FWIW, personally I found that doing a Linux (actually Solaris in my case, not Linux) sysadmin level course to be invaluable even for a developer. Not saying one needs to do that, but for me it helped a lot.

Plus learn about operating systems, and real-time OS's at least to a high level. There are books on this, many old, but a lot of the principles have not changed.

Get involved in a real project (come up with a use-case, even if it is contrived, and just start on it, and see the problems that occur, and see how you can measure, and how to solve them).

Also, consider that code running on Linux likely may be for a larger project, where you need to interact with other systems and processes too, and function with middleware and other components on-board and on the network; an excellent command of Linux and networking will come in useful for that too.

[betocool]

I did some work on Embedded Linux on an Altera FPGA/ARM core a few years back. The learning curve was steep, but I was able to customise my own build system after all.

If you find an existing image for your controller, life's good.

If not, life's still pretty good, but a bit more difficult. You can choose between Yocto and Buildroot to make your own build system. I tried Yocto for a week and could not get anywhere. I then had a look at a Buildroot example for my architecture and things started looking up.

There is an excellent article written by Jay Carlson about embedded linux: https://jaycarlson.net/embedded-linux/, but beware, the road is long and there be dragons. A great place to start.

I suppose with AI lurking around some things might be easier, but I haven't tried in a few years.

Hope that helps.

Cheers,

Alberto

[nctnico]

If not, life's still pretty good, but a bit more difficult. You can choose between Yocto and Buildroot to make your own build system.

Nooooo! If you like mental  pain or brain damage then use Yocto or buildroot. Realistically these are obsolete for 10+ years already. Nowadays you install Debian or Ubuntu on an embedded system. Using chrood and qemu you can run the installer on a host system which then builds a root filesystem for the target. Add the extra packages you need and done. The added benefit is that Debian or Ubuntu will supply security updates. A lot of dev boards come with Debian and typically include scripts to create a Debian root filesystem.

So what you need to bring up a board is a working bootloader tailored to the hardware and a kernel with the right drivers. Going from a development board, this is already done for you. So what is left is setting up the device tree to enable / disable certain drivers. For example when an SPI port and UART share the same pins, the device tree allows to select which peripheral to install. Also one thing to keep in mind is that the Linux kernel has a whole bunch of drivers for things like I2C expanders and it can also bit-bang interfaces like SPI or I2C.

Where it comes to writing device drivers: buy the book called 'Linux Device Drivers' published by O'reilly.

[PlainName]

You can choose between Yocto and Buildroot to make your own build system

I had to build Linux from source for a client, and every search for information resolved to 'use buildroot'. So buildroot was what everyone used and no-one seemed to actually know how to build linux from scratch without using buildroot. Circular bubble. (This is going back a while so I suspect that it will have got tighter since!)

Reason for wanting to do it from scratch was because the custom hardware used a PowerPC, needed custom device drivers, and had to be built under Windows Worked very well in the end, but buildroot made getting there harder than it should have been, despite not using it.

[EVblog1]

Thanks for the replies. so the discussion has drifted toward Buildroot, Yocto
At this stage, I'm trying to understand one specific concept before moving on to those topics:

I'm trying to build the right mental model of where kernel modules and device drivers fit in the Linux architecture before diving into the rest of the Embedded Linux ecosystem.

[nctnico]

Thanks for the replies. so the discussion has drifted toward Buildroot, Yocto

Als PlainName already implied: that is false information which keeps reverberating like in an echo chamber. Really, do yourself a big favour and stay clear from Yocto and buildroot. I have been dealing with embedded Linux for 20+ years already; take this advice from somebody who has been round the block a couple of times.

The kernel itself is pretty much a self containted entity. You cross compile it from the source code. The results are the kernel image, modules and kernel header files. You'll need the kernel header files if you want to build kernel modules (drivers) which aren't part of the kernel source.

[NorthGuy]

I'm trying to build the right mental model of where kernel modules and device drivers fit in the Linux architecture before diving into the rest of the Embedded Linux ecosystem.

OS provides common interface for userland software. Drivers translate this to real physical devices. This allows userland software to use abstract interfaces without knowing anything about physical devices. For example, I wrote a program 30 years ago when there was no LCD displays, no wireless keyboards, no SSD drives and so on. But my program still works without any changes because new drivers were written to support these new devices without need for any changes in my program.

Similarly, in your embedded Linux, if you write a serial linux driver for MCU UART then you can run any existing terminal software, use your UART as input/output for any program, run PPP and anything else which uses a serial port.

What you call drivers in MCU is just misuse of the word unless you have an OS with loadable apps.

[abeyer]

This book used to be one of the best sources https://lwn.net/Kernel/LDD3/

It's pretty dated in some of the details now, as it's been a long time since the last release, but it's available for free and a lot of the general background info is probably what you want anyway if you're going for "build the right mental model of where kernel modules and device drivers fit in the Linux architecture."

I'd start with skimming over the early parts of that first, to get that mental model established, and then you'll likely have a better sense of what you want to look for as far as the more modern details.

It's also not clear how much linux userspace programming experience you have, but if it's limited or rusty, might not hurt to also make sure you're solid on systems level userspace stuff, too, as that's the interface you're defining when building kernel stuff. There's more materials and options there... but you could do worse than https://nostarch.com/tlpi and https://nostarch.com/system-programming-linux if you're looking for a choice.

[EVblog1]

Yes, I actually have that book, and I'm using it as my starting point. I've also downloaded the Linux kernel source code and have started working through a simple character driver, as you can see in the image.


I understand that tools like Buildroot or Yocto can be used to create a custom Linux image. My confusion is earlier than that.

What I'm still trying to understand is what people mean when they say, "I wrote kernel code" or "I developed a Linux device driver."

In bare-metal development, if I want to connect a new device to an MCU and I don't already have support for the communication protocol (UART, SPI, I2C, etc.), I first write that reusable protocol driver. Then I use that driver to write my application. That's why I think of that protocol code as a "driver."

I'm trying to understand how that concept maps to Embedded Linux. How is a Linux device driver different from the reusable peripheral/protocol drivers that we typically write in bare-metal systems? That's the mental model I'm trying to build before moving on to Buildroot, Yocto, and the rest of the Linux ecosystem.

[hansd]

Linux is not a real time operating system. Linux or Unix in the olden days was running on a server with many terminals connected to. For example, if you use the POSIX's calls for serial communications it will not be processed in real time and there might or might not be a delay. To overcome this a kernel driver can be written to make the serial communication respond in a timely manner.

[PlainName]

How is a Linux device driver different from the reusable peripheral/protocol drivers that we typically write in bare-metal systems?

Do you write code that isn't for bare metal? Perhaps using an RTOS? Programming Linux or Windows apps?

[EVblog1]

How is a Linux device driver different from the reusable peripheral/protocol drivers that we typically write in bare-metal systems?

Do you write code that isn't for bare metal? Perhaps using an RTOS? Programming Linux or Windows apps?

Yes. I've spent some time learning FreeRTOS, but it wasn't for a specific commercial application. It was more of a hobby project to understand how the FreeRTOS APIs work—creating tasks, queues, semaphores, mutexes, and how they interact.

I've now started learning  Linux . I've downloaded and built the Linux kernel source code and have begun experimenting with a simple character driver to get familiar

[pcprogrammer]

I'm no expert on Linux, be it embedded or not, but for what I know of it, the basic idea is to make a separation between user and system, where the system can access every nook and cranny of the system and the user can not. This is where the device drivers come in to play. They give the user access to the hardware in a safe manner and restrict the user from damaging the system.

On a bog standard embedded bare metal system you most likely make no use of the processor provided system versus user space, at least I don't, and write "device drivers" in such a way that they make it easier for code higher up in the chain to not have to deal with the actual hardware. Allows for better portability of your higher up code.

Under Linux one has to be aware of the security and separation between user and system space when writing a device driver.

The kernel in itself is basically a task manager on which an operating system runs. The operating system is what provides a lot of the functionality, like windows, file access, etc.

You have to ask yourself what it is that you want from Linux on your embedded system, and what it gives you that is not easily available when running bare metal. I myself prefer bare metal because then you only have your own stuff, errors and all, to deal with.

Recently I spend a lot of time trying to get a Linux variant to run on a board I developed. Why, mainly to test the ethernet hardware on the board. That was in my eyes still the easiest way to perform the test, but it took a lot of effort to get the Linux (tina from Allwinner) build and running on my board. I used AI to guide me through a lot of the stuff, but that is also a pain in itself. But it is better than using google to search and distill the results yourself.

[Karel]

One disadvantage of using Linux (even RTOS!) is, is that it's never realtime!
I had this situation in a project where they moved away from using an mcu to a
system on a module (som) running Linux.
The spi bus was connected to an adc chip that generated a data ready signal every milli-second.
The new samples from the adc had to be read within 1 millisec so the data ready line was connected
to a gpio input of the som which triggered a kernel driver that read the new data and stored it in a buffer.
The os was struggling to keep up with the high interrupt rate and every now and then a sample was lost
which wasn't acceptable. In the end the solution was to put an mcu between the adc and the som in order
to create an "spi buffer" that easily read the data from the adc every millisec, buffered them and generated
a second data ready signal to the som every 32 millsec. Linux was then able to read the buffered data at a
much lower interrupt rate.
Linux does not support interrupt priorities and/or nested interrupts.

[pcprogrammer]

One disadvantage of using Linux (even RTOS!) is, is that it's never realtime!
I had this situation in a project where they moved away from using an mcu to a
system on a module (som) running Linux.
The spi bus was connected to an adc chip that generated a data ready signal every milli-second.
The new samples from the adc had to be read within 1 millisec so the data ready line was connected
to a gpio input of the som which triggered a kernel driver that read the new data and stored it in a buffer.
The os was struggling to keep up with the high interrupt rate and every now and then a sample was lost
which wasn't acceptable. In the end the solution was to put an mcu between the adc and the som in order
to create an "spi buffer" that easily read the data from the adc every millisec, buffered them and generated
a second data ready signal to the som every 32 millsec. Linux was then able to read the buffered data at a
much lower interrupt rate.
Linux does not support interrupt priorities and/or nested interrupts.

Wow, that is only 1kHz. Must have been a very slow SOM.

Several years back, I wrote a driver on the PC to handle isochronous USB data from a STM32F103 board and it had no problems with the 100kHz sample rate.

[EVblog1]

Linux is not a real time operating system.

I understand they're different. an RTOS is designed for time-critical applications where deterministic response times are important, while Linux is a general-purpose operating system. In a standard Linux system, there's no guarantee that a task will always complete within a specific deadline, whereas meeting timing constraints is one of the primary goals of an RTOS.

but right now my focus is on learning Embedded Linux and understanding its kernel architecture and device driver model.

[EVblog1]

So far, in my hobby projects, I've never had a situation where I actually needed to use Linux. Now that I want to learn Linux programming, I'm unsure about the best way to approach it.

One approach is to first understand why a particular kernel feature or mechanism exists before writing any code. I'm looking for real-world scenarios that justify kernel development. For example, writing a simple LED blinking kernel module helps me learn the APIs, but it doesn't really explain why the functionality belongs in the kernel instead of user space. Understanding that "why" gives me a much clearer mental model.

The second approach is more hands-on: write small kernel modules and simple drivers to become familiar with the kernel build process, APIs, and development workflow while gradually achieving small goals.

I'm not sure whether I should follow one approach or combine both. Personally, I prefer starting with the first approach because it helps me understand why a particular mechanism is needed and what limitations or problems would exist if it weren't used. Once that is clear, the implementation tends to make much more sense to me.

[nctnico]

The usual answer is: it depends. A project I'm currently working on uses a rather complicated chip. A Linux driver exists but that only exposes a fraction of the capabilities. Exposing all the capabilities through the driver would be a massive effort. The solution I'm going after is a hybrid one: a tiny kernel driver which catches interrupts and sets up DMA. All the rest is done from userspace by mapping the physical address space the configuration registers are at into userspace (virtual memory address). Handling the actual interrupts is done from userspace as well. A common pattern is to have the userspace program call the driver to wait for an interrupt. The kernel driver then waits for an interrupt to fire before it allows the userspace program to continue. This pattern is commonly used in videoplayers to synchronise playback to hardware. Wait for the vertical retrace interrupt before updating the screen contents (or flip the video buffer).

Actually the Linux kernel has quite a few features to mix kernel and userspace. The tap / tun interfaces for example which allow a userspace program to create a virtual network interface. A use case for this is VPN software.

[SpacedCowboy]

I went a slightly different route

The project here is to emulate an Atari 8-bit, and an Atari ST/TT on a single Zynq (ARM+FPGA in one chip). The zynq 7020's are now pretty "old tech" and they're easily available but they're still pretty "high tech" compared to 1980's computers To make it more authentic, I wanted a "switch on, <brief pause>, you're there", and I wanted control over the "brief pause" bit. Right now, I boot to a graphical desktop in ~2.5 secs, and 1.5 secs of that is my old 4K monitor syncing to the 1080p output.

The Zynq ships with two real options for an OS - linux or FreeRTOS, the easy option is just to go for linux, you get all the networking and display drivers built in, it "just works", but it fails my "switch on, <brief pause>, you're there" ready-to-go requirement. So I went with FreeRTOS

The basic idea is that you boot into a graphical desktop, can browse the filesystem (or Fujinet, a tftp-like internet-service, as if it were local) for 8-bit or 16-bit apps, and double-click to launch. That meant an underlying OS, hosted on the A9 core in the FPGA, vending services to the various emulators running as tasks (or hardware) - kind of like a hypervisor vending service out to the clients. There was one small wrinkle, I decided to make the A9 OS use an updated GEM as its graphical interface, which meant that "well behaved" 16-bit apps would just run on the desktop, not inside an emulator window.

The OS that eventually emerged from this was all built on top of the FreeRTOS kernel, extending it to allow:

• Shared libraries
• Dynamic linking of applications at runtime
• Memory protection
• Shared memory
• Virtual memory
• Networking, with BSD sockets
• Signals and job control
• A full /dev (block and character) and /proc interface
• Virtual filesystem interface (/tmp is ramfs, SD is fatfs)
  
• .. and probably a few things that escape me right now

On top of that base, I have sshd (dropbear) running and toybox (for unix-like utils) installed so I can ssh in/scp files, and there's a UDP mouse/keyboard redirect until the motherboard turns up for the Zynq module (a MyIR Z-Turn board) I'm using





It's actually pretty amazing what you can do with a little time and Claude code.

[langwadt]

One disadvantage of using Linux (even RTOS!) is, is that it's never realtime!
I had this situation in a project where they moved away from using an mcu to a
system on a module (som) running Linux.
The spi bus was connected to an adc chip that generated a data ready signal every milli-second.
The new samples from the adc had to be read within 1 millisec so the data ready line was connected
to a gpio input of the som which triggered a kernel driver that read the new data and stored it in a buffer.
The os was struggling to keep up with the high interrupt rate and every now and then a sample was lost
which wasn't acceptable. In the end the solution was to put an mcu between the adc and the som in order
to create an "spi buffer" that easily read the data from the adc every millisec, buffered them and generated
a second data ready signal to the som every 32 millsec. Linux was then able to read the buffered data at a
much lower interrupt rate.
Linux does not support interrupt priorities and/or nested interrupts.

Wow, that is only 1kHz. Must have been a very slow SOM.

Several years back, I wrote a driver on the PC to handle isochronous USB data from a STM32F103 board and it had no problems with the 100kHz sample rate.

USB is packets at 1kHz or 8kHz for full/high speed

[langwadt]

The usual answer is: it depends. A project I'm currently working on uses a rather complicated chip. A Linux driver exists but that only exposes a fraction of the capabilities. Exposing all the capabilities through the driver would be a massive effort. The solution I'm going after is a hybrid one: a tiny kernel driver which catches interrupts and sets up DMA. All the rest is done from userspace by mapping the physical address space the configuration registers are at into userspace (virtual memory address). Handling the actual interrupts is done from userspace as well. A common pattern is to have the userspace program call the driver to wait for an interrupt. The kernel driver then waits for an interrupt to fire before it allows the userspace program to continue. This pattern is commonly used in videoplayers to synchronise playback to hardware. Wait for the vertical retrace interrupt before updating the screen contents (or flip the video buffer).

Actually the Linux kernel has quite a few features to mix kernel and userspace. The tap / tun interfaces for example which allow a userspace program to create a virtual network interface. A use case for this is VPN software.

yeh, for accessing HW and interrupts  the UIO drivers go a long way

[pcprogrammer]

Wow, that is only 1kHz. Must have been a very slow SOM.

Several years back, I wrote a driver on the PC to handle isochronous USB data from a STM32F103 board and it had no problems with the 100kHz sample rate.

USB is packets at 1kHz or 8kHz for full/high speed

STM32F103 is only full speed, and yes the frame rate is 1ms, but the system still has to handle 16 bit data for 100kHz sample rate, which is way more than a single sample of 8 or 16 bits every milli second. Having to slow down the interrupt rate by 32 on a system capable of running some form of Linux is to me an indication that the system should not be running Linux in the first place.

The benefits of using Linux at times don't way up to the cons it brings.

[Karel]

One disadvantage of using Linux (even RTOS!) is, is that it's never realtime!
I had this situation in a project where they moved away from using an mcu to a
system on a module (som) running Linux.
The spi bus was connected to an adc chip that generated a data ready signal every milli-second.
The new samples from the adc had to be read within 1 millisec so the data ready line was connected
to a gpio input of the som which triggered a kernel driver that read the new data and stored it in a buffer.
The os was struggling to keep up with the high interrupt rate and every now and then a sample was lost
which wasn't acceptable. In the end the solution was to put an mcu between the adc and the som in order
to create an "spi buffer" that easily read the data from the adc every millisec, buffered them and generated
a second data ready signal to the som every 32 millsec. Linux was then able to read the buffered data at a
much lower interrupt rate.
Linux does not support interrupt priorities and/or nested interrupts.

Wow, that is only 1kHz. Must have been a very slow SOM.

Several years back, I wrote a driver on the PC to handle isochronous USB data from a STM32F103 board and it had no problems with the 100kHz sample rate.

If another interrupt (disk i/o, screen, whatever) arrives just before the interrupt related to the adc chip,
than the adc interrupt routine needs to wait until the other interrupt service routine has finished.
Hardware interrupts cannot be parallelized, they cannot be spread over multiple cores.
If that other interrupt takes 1 millisec, then there will be dataloss because the adc interrupt has not
been served before the new data arrives.
This is why many chips and (usb) chipsets for pc's have builtin fifo buffers.
Unfortunately, many adc chips don't have any storage for buffering.

[pcprogrammer]

Wow, that is only 1kHz. Must have been a very slow SOM.

Several years back, I wrote a driver on the PC to handle isochronous USB data from a STM32F103 board and it had no problems with the 100kHz sample rate.

If another interrupt (disk i/o, screen, whatever) arrives just before the interrupt related to the adc chip,
than the adc interrupt routine needs to wait until the other interrupt service routine has finished.
Hardware interrupts cannot be parallelized, they cannot be spread over multiple cores.
If that other interrupt takes 1 millisec, then there will be dataloss because the adc interrupt has not
been served before the new data arrives.
This is why many chips and (usb) chipsets for pc's have builtin fifo buffers.
Unfortunately, many adc chips don't have any storage for buffering.

Again true, but with prioritizing of interrupts it could be arranged that the data keeps coming in. If an interrupt routine needs such a long time to process, it kinda is a bad piece of code.

But that is one of the reasons I like to do things bare metal, full control over every part of the system and no one to blame but yourself when things do not work. There is a lot of crap code out there