From Bare-Metal to Embedded Linux

[Tation]

I found using buildroot not rewarding at all as a learning tool. After configuring tons of options, you finally obtain a disk image to flash, so what? Just boring.

Writing a driver is much rewarding, IMHO, you need to understand what the kernel expects from you and how to use the services it provides to you. And you only need a box running Linux, a device to manage, the kernel headers, a compiler and a book or some tutorials. Also, feedback is faster, as testing each try is faster.

In any case, maybe the OP can clarify his aim. I do not yet know if he wants to learn the kernel workings, or porting Linux to a yet unsupported board, or add support for a not yet supported device, or create a Linux distro tailored to some product or what.

[paulca]

I could suggest that making an embedded linux from scratch really needs to be done from source on the platform in question.

You can cross compile it, but as a beginner I would not recommend it.

A root builder kit for a distro... honestly, I would avoid it.  You'll end up with a custom kernel and custom Initrd file and a whole mish-mash of dynamic modules you don't need and don't want.

You would be better off building your own kernel and your own rootfs, from source.

Configuring and building a vanilla kernel from kernel.org is a BRILLIANT starting place for you.  Even if you don't go "build from source" the kernel is almost always the exception.  NOT the Ubunutu kernel!  Seriously.  A vanilla kernel.

Building a rootfs from source however can be a bit more tricky.  If the target dev board has got a bit of snot to it, then it should be capable of compiling itself.  This is the purest form of source build and you only need a single "bootstrap" compile of the compiler.  If you want to compile on host A to run on host B it gets more complicated with things like CPU Architecture.

The reason you want to build from source is that each of the basic rootfs packages have literally dozens and dozens of compile time options.  Each of them will increase the dependency graphs and the memory/binary size/filesize demands.  Being in the place where you "choose" if bash is even installed and if so if it has "readline" or "autocomplete" support.  Is important if you have a small footprint as a target.

[EVblog1]

In any case, maybe the OP can clarify his aim. I do not yet know if he wants to learn the kernel workings, or porting Linux to a yet unsupported board, or add support for a not yet supported device, or create a Linux distro tailored to some product or what.

My primary goal is to gain hands-on experience with Linux application development, kernel programming, and device driver development. I've already started this journey using a Raspberry Pi.

At this stage, my priority is to understand the overall development process, how an embedded Linux system is built, how applications interact with drivers, how drivers interact with the kernel, and how everything fits together. Once I have that practical understanding, I'll go deeper into Linux internals.

I'd also like to build a small custom Linux image using Buildroot not because I think Buildroot itself teaches kernel internals, but because I want hands-on experience creating a minimal embedded Linux system

[paulca]

On "Why OS?  Why Kernel?"

I have been on this journey a while now.  From Z80 breadboard and approaching 68000 computer... with custom rolled OS (Roms on the z80).

The very first thing you need to do on a bare CPU like the Z80 is give it an application to run.  The way you do that is typically via a command prompt, "LOAD """ for instance in Sinclair basic.  That would then take a stream from a tape device, a serial chip of some kind.  Load it into memory and then "RUN" it.   Boot it back up and load a different program etc.

"It's programmable"

In some cases you could exit the program back to the command prompt (or relaunch the command prompt) and LOAD and RUN a new program, right?

This parallels with the early "mainframe" computer days in the 60s.  People where physically shuffling around tapes and punch cards and... an "Operator" who orchestrated the actual computer, pointing it at the right tapes, pointing it at the right line printer, punch card reader and then start it loading the program from tape or card, then running it once the data IO tapes/cards are ready.  When it was done, it was all reset back for the next job.

IBM I believe were the first to automate that part and call it "Operating system".  Now jobs could be sent in off cards or tape themselves.  "Load this program on tape 3, make IO1 and IO4 available and log output to line printer 3."  Various job control languages spun up off this.

This is "multi-programmable" with "automated operating system".

The next big paradigm changes where in the form of "Memory gets cheap" and "Multi-user", "Multi-tasking" is being demanding by business.  This is where the "Kernel" thing starts to come in, mostly from Unix.  The separate, arbitrator over managing access between multiple users and finite system resources in a sensible way.  In this model the "application" code does not call hardware at all, ever.  It MUST call via the kernel.  The kernel is the only process which can assert the "HARDWARE" "supervisor" flag and control it.  All applications must ask via the kernel.

When Virtual Memory came around, full inter-process memory segmentation/paging became a thing and the kernel manages the hardware MMU to prevent process A from accessing process B's memory pages.  Any memory allocation, etc.  MUST be done via thekernel.

Tying this back to your OG question about "Why kernel in embedded project?"

Its going to be the same questions as above.

Do you have multiple processes?  Do they all trust each other?  Do they share access to things and need an arbatrator?  What are my IPC security requirements?  How do I want to managage dynamic memory? 

But also, as you are taking an OS "off the shelf" you will ask:

Do I need complex driver or firmware features like a network stack, filesystems, et. al.

Usually these are baked in to the kernel.

if you intend to run a single process, present no shell, don't need a network stack or filesystems, then you probably don't want a kernel or an OS and can stick with mbed or RTOS.

On RTOS.  A typical kernel is NOT RT.  It's NRT.  Near real time.  Almost all kernels operate on an internal "tick" scheduler.  Ticks can be as course as 100Hz.  If you need closer to real time, you need at least a "Pre-emptive kernel" which allows the kernel to interrupt itself to serve hardware/io interrupts.  Used in "Audio builds" on Linux.

So the penalty you pay for having the kernel is never truely being in control of the hardware and not even really in "lock step" it it's heartbeat/timing either.  You have to deligate that to the kernel.

There are caveats of course.  You can expose as much or as little hardware control as you like.  You will run into issues with the way kernels handle certain event structures etc, but it's plausible to get "near raw" or "barely off the PCIe bus" control.  Just not very easy due to timing constraints.

[EVblog1]

You would be better off building your own kernel and your own rootfs, from source.

I've already downloaded the Linux kernel source, compiled it, and successfully built and loaded a simple "Hello, World!" kernel module. I'm not claiming that I've achieved anything significant, but it helped me understand what a kernel module is, how to write one, create a Makefile, and load and unload it from the kernel.

Now I'd like to follow the same learning approach with device drivers. My goal is to go through the complete driver development process from writing the driver to testing and debugging it. and that's what I'm currently working on.

I'd also like to take the same approach to building a Linux image.  I want to understand the process, what each component does, why it's needed, and how everything fits together. My goal is to gain hands-on experience with the complete workflow

I also feel that once we have a working setup, it's much easier to experiment with it. We can intentionally change or remove certain components and observe the effects. That helps me verify my understanding and build confidence in how the system actually works.

[BradC]

I'd also like to build a small custom Linux image using Buildroot not because I think Buildroot itself teaches kernel internals, but because I want hands-on experience creating a minimal embedded Linux system

25 years ago I was building linux images with X and Wine to run Delphi apps on an embedded PC in RAM loading from a 16M CF card. Learning to compile and assemble minimal linux systems for fairly constrained scenarios really taught me a lot about how it all goes together and the minimalist approach. I used a pile of hand written scripts to make a reproducible environment. Buildroot is pretty much a slightly more modern version of that and a pretty good place to start.

These days with machines with bountiful RAM, you can run an entire OS from the initramfs. Grab busybox and crash-and-burn your way to a basic minimal embedded OS. With Qemu on a desktop, burn and boot cycles can be seconds rather than minutes on real hardware and you can make rapid progress.

Heck, grab a debian live-cd and boot with init=/bin/bash and bring a system up from scratch. That'll give you a decent idea as to what you really need vs the bloatware that is a "modern" linux distribution.

[Nominal Animal]

Suppose we have a Linux-based embedded device whose only job is to read data from sensors and publish that data to a cloud server over MQTT. Since the device is designed for this single, dedicated purpose, my understanding is that we don't need a full general-purpose Linux distribution.

In my opinion, the best approach is to treat it as a fully featured server, because it will be exposed to the internet (even if only "slightly").

(It is obviously possible to do it on a suitable microcontroller, say leveraging mongoose, but I don't like the idea of trying to secure that against the security maelstorm that internet is.)

For a server (including NAS boxes), starting with minimal Debian install is a pretty good idea.  Even if it is technically an "embedded device" or "appliance", I do believe these are best (security and reliability-wise) treated as plain Linux servers exposed to the internet.  From minimal Debian,  on top of the base system (filesystems, base util-linux stuff), you'll get a robust networking support (WiFi or Ethernet, DHCP client, DNS caching) et cetera.  With WiFi or 4G/5G/LTE networking, you'll also want NetworkManager (and ModemManager, if you use an USB-connected 4G/5G/LTE module).  You can use systemd for ease of maintenance: on a headless server with very few services and no interactive humans, it tends to do well.  You will need TLS support for the data-gathering-and-publishing daemon; choose between openssl, gnutls, mbedtls at least in Debian.

Since the data bandwidth is relatively low, you do not necessarily need to write the service in a compiled language like C, C++, Rust, etc., but can also use interpreted languages like Python.

I would focus here on server-side development, making the service robust, including testing it with severely limited run-time limits, executing your service daemon using prlimit -m 200M yourdaemon... . Examine how it behaves when it starts running out of memory in particular, and how it handles loss of Internet access (including changes in local IP address then).  And always test what happens when it is bombarded with fake connection attempts and such.  You can simulate all that using local fixed IP addresses, and bringing up and taking down net interfaces using ip (iproute2 package) or ifconfig (net-tools package), and using netcat or socat (many alternatives) from a Bash one-liner loop to fake the connections.


One important point on Linux server side development, be it embedded or fully-featured: you must read up on POSIX.1, and its system interfaces in particular, because that is what your C library (glibc, newlibc) your entire userspace will use already implements.  The old K&R C, or the C11 or later as implemented by e.g. Microsoft, is a wooden mallet compared to POSIX C when it comes to networking and server-side software development.  Even if your SBC/SoC has only one physical CPU core, using pthreads sensibly (with suitably small stacks in particular), you can get more out of that core; the per-pthread overhead in Linux (aside from default stack size, which tends to be huge, allocated for each thread) is very, very low.

Just as an example, you'll want to be able to use getline(), nftw(), glob(), fnmatch(), regcomp()/regexec()/regfree(), POSIX signals (and use sigaction(), never signal() or sigvec()), gettext() for internationalization/language versioning, iconv_open()+iconv() for character set conversions if necessary; and so on.  Otherwise, you'll be wasting your time reinventing a poor wheel while a time-tested, reliable one is already included in the C library your userspace is based on.  (Applies to both glibc and newlib, by the way.)

[paulca]

Suppose we have a Linux-based embedded device whose only job is to read data from sensors and publish that data to a cloud server over MQTT. Since the device is designed for this single, dedicated purpose, my understanding is that we don't need a full general-purpose Linux distribution.

In my opinion, the best approach is to treat it as a fully featured server, because it will be exposed to the internet (even if only "slightly").
.)

If it's an MQTT client to the cloud it is active connection only and happy behind a NAT or firewall and "safe enough for consumer".

Where the trouble will lie is on the "cloud" end.  If it's an actual cloud like AWS/Azure it will present you with the latest, greatest set of encryption and auth challenges and web security, because it's the risky end.

This will knock down into your MCU layer.  The crux point will be what type of SSL/TLS and certs it uses, how much crypto processing your devices has to do on what sized artefacts etc.  Also, how dynamic are the keys and if outside of your "Field maintainence" practicality, a way for the devices to renew their own keys.

You would be surprised how quickly all that adds up.  While bare bones ESP32 or ESP8266 come from factory able to connect to MQTT with basical encryption, IIIRC.  I would guess that AWS/Azure are going to make that a lot more complicated and you might need a proper MCU like an STM32H7...

... or embedded linux with full top of the line, latest and greatest SSL libs,

The halfway house is using one of AWS or AZures MCU devkits designed for the purpose and just weep at their hardware requirements.

[paulca]

One more small point on "Embedded Linux" specifically.

Linux has certain features which make certain hardware demands, such as actual virtual addressing and MMU.  Without this Linux is not really linux and the flat and segmented memory models are not that well maintained anymore.

That is absolutely core to how modern computers work.  I don't even think the STM32H7 class can run "native linux".  Even the 68000 CPU had issues with a hardware MMU in tandem because it can't restart instructions after a  BERR / Page fault.

Yes, embedded linuxes exist in communities where linux is forced to run without a full VM and MMU, but it starts you on a road of stripping away enough of the OS that it stops being that OS anymore.

[EVblog1]

For now, I think I'll put the server and MQTT work on hold for a while and focus more on Linux device driver development process.

I'd really appreciate your thoughts on my comments in posts #77 and #79. Do you think I'm heading in the right direction, or would you recommend a different approach?

[paulca]

For now, I think I'll put the server and MQTT work on hold for a while and focus more on Linux device driver development process.

I'd really appreciate your thoughts on my comments in posts #77 and #79. Do you think I'm heading in the right direction, or would you recommend a different approach?

You have a raspberry PI.  I personally would start there by downloading the vanila kernel  and compile it.  Start with "just" compiling it. DO NOT install it on your Raspian or debian!  Doing so is likely to cause boot failure.

Compiling the kernel teaches you a LOT of about how the kernel is structured, which bits hang off which other bits etc.

Compiling the kernel "proves you build chain tooling etc."

Compiling the kernel is probably the single "least dependant" compile you will ever do on *nix.  The kernel depends on nearly "FA squared".  It does have some requirements for it's build tool chain though.

If you do go to your first "Hello World" driver it will be via this same process.  You write your module in the correct folders structure, provide the correct meta-data for it and the "make menuconfig" will list it as a module.  You tick it as "Built in" or "Module" and build the kernel.  Thats the only way to get a kernel driver (properly).  It's why distro's leave the source in place and the headers installed for the kernel, so that device "installers" can actually install their device drivers requires as a minimum the "headers for the running kernel".  Not just any old kernel.  The running one.  Headers are dynamic.

There will be tutorials out there on hardware drivers.  Where I would be careful though is, "Has it already been done before?", or "Is there a higher level abstraction I can just reuse?".  For example.  Most sensor bus chips, uarts, i2cs, GPIO stuff is already presented by drivers in the kernel.  Sometimes starting with one of those is enough. 

A GPIO level driver which allow syou to "bit mash" some peripherals like an LCD module of AliExpress might be intstructive.

Its been a while since I did a kernel, but the process was just:

wget ... kernel url
cd kernel source
make clean (used to be mrproper)
cat /proc/config.gz > .config && make oldconfig - if you have a working config.  "make menuconfig" if you don't.  You can hybrid them by writing the .config from your existing config then run make menuconfig to modify it.
make all
make modules
#make install
#make modules_install

[paulca]

The RPI is a good canidate for this exploration too.  Its got enough snot to complie it's own OS, so won't take an hour to do a kernel.

If you go smaller you will end up in a place where the host cannot compile it's own kernel, then ... things get complicated with cross compiling.  Cross compiling kernels requires a cross compiler for the target and a full set of headers and dependencies also compiled for that target... including the compiler itself. 

So at that point stay with "supported archs" which already have cross compilers easy to access.  Compiling things like kernels and "glibc" or other core dependencies can get a bit dodgy IIRC.  When I was doing this, it was way back before distro's shipped with a bunch of cross compilers and you had to boot strap your own.  It's not fun.  Bootstrapping even i486 on an i686 took me days of effort to figure out how to isolate the compiler and convince it to bootstrap for an arch it was not running on.

[nctnico]

At the moment, I don't have a real hardware problem that requires me to develop a brand-new Linux driver from scratch.

Would it be a reasonable learning approach to intentionally remove or disable an existing driver and then try to get the hardware working again by developing or fixing the driver myself? My goal isn't to create an artificial project, but to gain hands-on experience with the complete driver development and debugging process.

One of the things I did for a recent project is change the Linux I2C eeprom driver so it can be used with hot-swappable devices. The regular driver probes the I2C eeprom device at startup and doesn't register it if it isn't present. What I did was add a device tree flag to ignore the probe step and register the device either way if the flag is set. That could be an interesting exercize.

[EVblog1]

One of the things I did for a recent project is change the Linux I2C eeprom driver so it can be used with hot-swappable devices. The regular driver probes the I2C eeprom device at startup and doesn't register it if it isn't present. What I did was add a device tree flag to ignore the probe step and register the device either way if the flag is set. That could be an interesting exercize.

do you want me to implement this exercise on my Raspberry Pi?  so I can modify the AT24 EEPROM driver, update the Device Tree, rebuild the kernel/module, and test the changes. I just want to make sure that's what you had in mind.

[nctnico]

One of the things I did for a recent project is change the Linux I2C eeprom driver so it can be used with hot-swappable devices. The regular driver probes the I2C eeprom device at startup and doesn't register it if it isn't present. What I did was add a device tree flag to ignore the probe step and register the device either way if the flag is set. That could be an interesting exercize.

do you want me to implement this exercise on my Raspberry Pi?  so I can modify the AT24 EEPROM driver, update the Device Tree, rebuild the kernel/module, and test the changes. I just want to make sure that's what you had in mind.

That is what I did indeed. BTW it is only suggestion but doing this takes you through device tree settings and modifying a driver

[EVblog1]

I've tried blinking an LED in two different ways. First, I wrote a user-space application that uses system calls to control the LED. Then I wrote a kernel-space driver to do the same thing and it works

From that, it seems like I could follow the same pattern for other things as well, for example, reading a button, controlling an ADC, reading a temperature sensor, or communicating over SPI or I2C devices. In other words, it looks like I can achieve the same goal from either user space or kernel space.

This is where I need opinion .

How do you decide which approach to use? What makes you say, "This should be a user-space application," versus "This really needs to be implemented in the kernel"? How do you make that decision in a real project?

[paulca]

Timing and control.  Permissions.  Usually. 

Can a user aka, non-root access your blinking LED via a standard /dev/blinky with normal user permissions and then write to the correct mapped port or whatever?

If it was an SPI module, what happens when the user application is suspended pre-emptively?

It's usually not a choice.  You simply cannot write directly to hardware addresses without the kernel either providing access or accessing it in proxy for you.

What would /dev/blinky support?

You could accept "1 or 0" for on or off, but that might live in /sys/ or /proc/ interfaces instead.

You could take a struct with frequency and duty cycle.

[paulca]

"Bit banging SPI" on a macro-kernel might be an interesting and revealing task to attempt mind.

There are existing SPI drives in the kernel though and lots of support for hardware SPI.

[Siwastaja]

I've tried blinking an LED in two different ways. First, I wrote a user-space application that uses system calls to control the LED.

This is where I need opinion .

How do you decide which approach to use?

If you think deeper, there's a small mispresentation of what you say. You really don't have two different ways as "control LED from userspace" vs. "control LED from kernel" - in both of your experiments, the kernel was turning the LED on and off!

So what was really different is, who wrote the kernel module and what features did it contain?

And that answers your question. Your "from userland" experiment used some existing GPIO driver wrote by some other. You controlled that module from userspace, but so you would control your own module. So, really the same thing! If the existing GPIO module offers the right features for you, then you don't need to write your own.

What could be those right features? Usually related to timing and permissions like paulca said. Say, you want your LED to be ON for as close to 10 microseconds as possible. Maybe the GPIO driver you tried cannot do that. Maybe it only offers "turn on" and "turn off" commands, so your userland application is then: turn_on(); usleep(10); turn_off(), but delay being in userland, it's not going to be as accurate. But if you write your own kernel driver, you can use the timing features kernel offers - even interrupts - and then expose a new function pulse_on(uint64_t nanoseconds) which you can now call from userland. Or something like this.

But the actual HW access is always from kernel, so not really "two different ways" in that sense.

[nctnico]

But the actual HW access is always from kernel, so not really "two different ways" in that sense.

That doesn't have to be the case. If you map the physical address space of the registers which control the GPIO pins into the virtual memory space of your program, you can access hardware directly. Without the kernel in between.

[pqass]

How do you decide which approach to use? What makes you say, "This should be a user-space application," versus "This really needs to be implemented in the kernel"? How do you make that decision in a real project?

Generally...

Hardware must be protected from direct access by any random program to enforce security (eg. to keep a shared disk segregated by user) and to not interfere with the kernel code that manages it (eg. two cooks grabbing the same pot).

However, if the hardware isn't being managed by (or unknown to) the kernel, then you could tell the kernel to expose the iospace and use inb()/outb() to interact with the hardware directly as a user space process.  See below for an example.  I'm not sure how to expose memory-mapped io.

Code: [Select]

#include <stdio.h>
#include <stdlib.h>
#include <sys/io.h>

// $ gcc -O -o iotest iotest.c
// $ ./iotest
// ioperm(3f8) = -1
// $ sudo sh
// [sudo] password for pqass:
// # ./iotest
// ioperm(3f8) = 0
// inp(3f8+0) = 32
// inp(3f8+1) = 0
// inp(3f8+2) = 1
// inp(3f8+3) = 0
// inp(3f8+4) = 0
// inp(3f8+5) = 60
// inp(3f8+6) = 0
// inp(3f8+7) = a0
//
// I'm not sure why it has to run as root vs chmod ugo+s (setuid) not being good enough

int main() {
  int r;
  int a = 0x3f8; // ioaddr of first serial port
  printf("ioperm(%x) = %d\n", a, (r = ioperm(a, a + 8, 1)));
  if(r == 0)
    for(int i = 0; i < 8; i++)
      printf("inp(%x+%x) = %x\n", a, i, (int)inb(a+i));
  exit(0);
}

But, as a rule, code running in the kernel (drivers) should be minimized because bad code will crash the machine.  The more code running in the kernel, the greater likelihood of bugs, and thus higher probability of a crash.

Also, I don't think the kernel thread is interruptible by other processes (must explicitly yield) so doing too much holds up every other process on the CPU.  Optimally, one should just interact with the hardware enough to move data to/from memory then do any other processing in user space.  In user space, your code can be interrupted (context switching), fail and won't crash the machine.

[EVblog1]

I think I had a misunderstanding. When I used to hear people say, "We don't expose the hardware to user space", I assumed that my user-space application shouldn't know things like which GPIO pin the external LED was connected to. I thought all of that had to be hidden behind the Device Tree or a kernel driver.

But after trying it myself, I realized that's not necessarily true. In my case, I still had to tell my user-space program which GPIO line the LED was connected to.

So I think I misunderstood what people mean when they say, "We don't expose the hardware." It seems that phrase means something different from what I originally thought.

[nctnico]

I think I had a misunderstanding. When I used to hear people say, "We don't expose the hardware to user space", I assumed that my user-space application shouldn't know things like which GPIO pin the external LED was connected to. I thought all of that had to be hidden behind the Device Tree or a kernel driver.

But after trying it myself, I realized that's not necessarily true. In my case, I still had to tell my user-space program which GPIO line the LED was connected to.

So I think I misunderstood what people mean when they say, "We don't expose the hardware." It seems that phrase means something different from what I originally thought.

IMHO you should start reading books about these things instead of trying to pick up of snippets of information without getting the big picture. A book tells you a coherent story while painting the big picture and takes you through the pros & cons of solutions and which solutions are applied when and why.

[ejeffrey]

I think I had a misunderstanding. When I used to hear people say, "We don't expose the hardware to user space", I assumed that my user-space application shouldn't know things like which GPIO pin the external LED was connected to. I thought all of that had to be hidden behind the Device Tree or a kernel driver.

Yes.  You can go to this extent, and sometimes it makes sense.  But it's not needed.  The main point of the kernel driver is to protect the actual IO memory from userspace -- so you use a write to /dev/gpiochip4 instead of writing to physical memory address 0xC040_0000 or wherever to access the GPIO controller's physical registers.

Using the device tree to add aliases is nice if you have something like a SBC and you want to name the GPIOs to match the buttons on your PCB.  This also gives you some flexibility to respin the board with different pin assignments, and you just rev a device tree overlay.

[EVblog1]

The main point of the kernel driver is to protect the actual IO memory from userspace -- so you use a write to /dev/gpiochip4 instead of writing to physical memory address 0xC040_0000 or wherever to access the GPIO controller's physical registers.

That makes sense now.

So the idea is that a user-space application should not access the GPIO registers directly. The kernel is responsible for protecting the hardware registers from user space. Instead, the user program makes a system call, and the kernel accesses the GPIO registers on the application's behalf.