From Bare-Metal to Embedded Linux

[paulca]

Using the device tree to add aliases is nice if you have something like a SBC and you want to name the GPIOs to match the buttons on your PCB.  This also gives you some flexibility to respin the board with different pin assignments, and you just rev a device tree overlay.

This is one of the areas where an OS like Linux plays a bit of "polymorphism".  You can have multiple device files point at your "device driver" and your device driver knows the name of device file you are using.  Thus it can select it's process/operation based on that.

Its similar to how "run as" works.  Take "Busybox" for example.  It's a single binary which hold dozens and dozens of applications.  All you do is symlink "cp -> busybox" and now "cp" behaves like.... cp does.  If you also symlink it as "grep" it will behave like grep.

[paulca]

On hardware isolation via the kernel.

In OS design, "Unix-like" version.  Memory is split.  You have a "Virtual address space" and you have "Physical address spaceS".

Every application gets a full, empty, virtual address space for themselves.  They can all write to address 0x0000_0000 and you can run 10 of them an they will be fine.

So where IS 0x0000_0000 really?  Mapped via an MMU TLB entry to a page of RAM in one of the Physical address spaces.

Note.  Mapped in HARDWARE.  There is no way, at all, for you to get to the "real" address 0x0.  It doesn't exist.  Not until there is a TLB (Table lookup buffer) entry in the look aside tables for the MMU on the motherboard/chipset/CPU.

Each process gets it's own set of TLB entries.  They include things like:

Pages of kernel shared memory.  Used for calling kernel structures and referencing kernel structures considered "public".
Your actual memory pages.
Other shared pages for IPC.
Mapped direct IO pages
Stack, cache, buffers etc. etc.

Upon your process spawning and being setup, these mappings will be created.  When you process moves to "Running", the scheduler will evict the TLB entries for the prior process, write the new mappings for your process and then set the PC to your entry point.

When you then call

putchar( (char*)0x0000000 );

That address 0x00000000 maps to some page of physcial memory at a different pyhsical address.  Like 0xF00C0000

Depending on era, most or ALL of the virtual address lines go via the MMU and get re-rewitten.

If you do not allocate memory and try and access it, the Linux "Segmentation fault" is a misnomer.  It retains it's OG name even though it's purpose and truth changed.  It really means "Page fault".  Your "Virtual address" scanned the TLB and came back with a "Not found".  This results in a page fault or a "Address violation".

Devices/hardware/buses are almost always mapped to the address space.  Pyshical ones.  If your process wants to access the GPIO port in physical memory space it will require a TLB mapping inserted.

The MMU/TLBs are ... again... HARDWARE controlled regards to access.  There are various status flags set on the CPU which are sent to the MMU preventing the user for accessing the MMU itself.  So you can't "add your own" mappings.  You MUST ask the kernel.

Of course.  It is possible and in some instances valid to just give the user process a memory mapping for the device and let them tear away.

What you would want to assess though is...  can that user appliction do anything with that device directly which could compromise the stability of the system for the other 33 users?

[gf]

So where IS 0x0000_0000 really?  Mapped via an MMU TLB entry to a page of RAM in one of the Physical address spaces.

On modern Unix-like operating systems, virtual address 0 of a process's user address space is usually unmapped, in order to enforce a segmentation fault when a NULL pointer is dereferenced.

[paulca]

So where IS 0x0000_0000 really?  Mapped via an MMU TLB entry to a page of RAM in one of the Physical address spaces.

On modern Unix-like operating systems, virtual address 0 of a process's user address space is usually unmapped, in order to enforce a segmentation fault when a NULL pointer is dereferenced.

As an asides.  GCC itself will remove pointer access to address 0.  I found that out the hard way when working with the 68k arch on baremetal.  I wanted to test the ASM "bootloader" had correctly mapped RAM to address 0 (ROM on boot).  So I do a write/read on address 0 in a test harness.

The code ran.  Exited and failed the test.  It took me a long time to find out and only by disassembling the binary did I find that GCC had literally just removed all the code which accessed address 0.  It likely did leave a compiler warning I ignored.

I had to enable the -embedded mode for it to permit me to address 0.  This is documented and often architecture specific.

EDIT:  "Physcial" addresses here, not mapped.  Direct CPU address bus to RAM IC.

[gf]

So where IS 0x0000_0000 really?  Mapped via an MMU TLB entry to a page of RAM in one of the Physical address spaces.

On modern Unix-like operating systems, virtual address 0 of a process's user address space is usually unmapped, in order to enforce a segmentation fault when a NULL pointer is dereferenced.

As an asides.  GCC itself will remove pointer access to address 0.  I found that out the hard way when working with the 68k arch on baremetal.  I wanted to test the ASM "bootloader" had correctly mapped RAM to address 0 (ROM on boot).  So I do a write/read on address 0 in a test harness.

The code ran.  Exited and failed the test.  It took me a long time to find out and only by disassembling the binary did I find that GCC had literally just removed all the code which accessed address 0.  It likely did leave a compiler warning I ignored.

I had to enable the -embedded mode for it to permit me to address 0.  This is documented and often architecture specific.

EDIT:  "Physcial" addresses here, not mapped.  Direct CPU address bus to RAM IC.

I think you can use the -fno-delete-null-pointer-checks flag to force GCC to keep actual memory-access instructions and null-pointer validation checks, instead of optimizing away paths that the compiler assumes are unreachable due to a null pointer dereference.

[EVblog1]

I now have a high-level understanding of Linux device drivers and user-space applications, and I'm currently working on both. To understand Buildroot, I built my first minimal Linux image for a Raspberry Pi 3. I configured Buildroot, generated the Linux kernel, root filesystem, and a bootable sdcard.img. At the moment, the image only contains the Linux kernel, BusyBox, and a minimal root filesystem.



My goal is to understand how Buildroot is actually used in a real embedded Linux project. I already have a simple LED driver and a user-space LED application, and I'd like to use them as my learning project instead of creating a new example.

I understand the process of creating a minimal image, but I'm not sure about the next step. I want to understand the workflow used in real embedded Linux projects to integrate an existing application into Buildroot.

[paulca]

You have a very weird way of asking questions from odd angles while tracking things in bizzare orders.

What is the purpose of buildroot?

Have you see the size of Debian vanila?  You did this with a RaspberryPI 3.  Try it with a i386 with 64K of RAM or on some "nano scale RISC-V thing"  Then you will understand why build root.

There are embedded linux systems inside things like "POS terminals" which run on 68k architecture and have a few dozen kilobytes of RAM.

If you have an RPI you don't need build root it would be a waste of time and effort, not least because the PI is not an embedded platform.

and that is the more direct answer to your question.

buildroot gets you to a "fully operating general purpose OS".  After that you are no longer in embedded world.

"Embedded linux" literally just means "Linux embedded inside something other than a PC"

Bank ATM terminals would also be a "miss match".  They are not embedded.  Nor are kiosks.  Nor are industrial control rigs running on PCs.

Once you have a working linux base system it stops being embedded.  You are in the wrong forum at that stage.  The whole point of going that mile and funding that hardware is to get away from the embedded restrictions and into the comfort of an actual OS.

So your tooling moves to "apt" or whatever packaing system your choosen buildroot wants to use.  If you want a driver, install it.  If you want to write it, write it. Just be careful about the cross compiling.

The bigger decision is often "How much runtime do you need to support".  The fully fledged glibc is MASSIVE.  Striping it down by dumping internationalisation etc still results in a "many megabyte" sized binary library.  Different levels of "minimising" are available with forks, but eventually you start even stripping out parts of "stdlib".

The majority of Wifi and Internet routers these days run a Linux kernel and present a web interface as a service.  Underneath it, it's Linux kernel structures and drivers running the show.  The network stacks, wireless drivers everthing with a pretty web UI either as it's "only shell process" or as an auto started one.  "Advanced routers" also provide a log in command line shell.

So pick any old router in your house and investigate it.  Notice how when the Web UI crashes or locks up the router still "routes".  Guess where each lives in kernel/user space.  "Open WRT" is a prime example of an "arugably" embedded linux.  Runs out of about 32Mb of EMCA flash usually.  Will your "root build" fit in that?

[EVblog1]

You have a very weird way of asking questions from odd angles while tracking things in bizzare orders.

My objective is a little different. I want to learn Buildroot itself. I did the Buildroot work on my Ubuntu host and I'm using the Raspberry Pi only as the target platform to understand the workflow.

I already have a simple LED driver and a user-space LED application. My goal isn't to blink an LED using Buildroot; I'm using that project as a simple example to understand how an existing application is integrated into a Buildroot project so that it is automatically cross-compiled and included in the generated image.

Once I understand that workflow, I can apply the same approach to larger embedded Linux projects.

[paulca]

You have a very weird way of asking questions from odd angles while tracking things in bizzare orders.

My objective is a little different. I want to learn Buildroot itself. I did the Buildroot work on my Ubuntu host and I'm using the Raspberry Pi only as the target platform to understand the workflow.

I already have a simple LED driver and a user-space LED application. My goal isn't to blink an LED using Buildroot; I'm using that project as a simple example to understand how an existing application is integrated into a Buildroot project so that it is automatically cross-compiled and included in the generated image.

Once I understand that workflow, I can apply the same approach to larger embedded Linux projects.

The standard interface separation is via the /dev/ /sys/ /proc/ hierarchies and direct kernel traps if you need.

So the C application, if you choose to use those standard "everything is a file" interfaces just needs basic "Stdlib file IO" and nothing special.

In theory you should be able to compile it on any linux and run it and have it say something like "/dev/gpio" not found and exit.

Take that application and 'cross compile' it for the architecture in question and the only remaining hurdle is "linkage".  When your binary calls the symbol for "getchar" you need to be sure it hits the right symbol in the library table.

Here my experience runs thin.  In my prior works here I have always choosen and been able to take a "Parent" self building root FS and then prune it back to an installable root "minus" all the debug symbols and cruft that comes with compiling a root FS.  So I never "cross compiled".  I just surogated the build host.

As a funny aside... that system which "spawned Linux systems" from source, was actually called "Mother" by me.  When I wanted to change an OS/root FS, i would update the config and run the build overnight using "Mother" and the next morning I would find a bootable Linux, all I had to do was swap grub entries.

[paulca]

I should point you to one very common trick using in embedded and smaller scale single purpose builds.

You will note the kernel takes a parameter of what to hand over to after it has booted and iniited.

"init" process.

Usually this is set to something like /sbin/init which starts the whole "RC boot" process, either SYSV or SystemD.

You can set it to whatever you like.  It will be the first userspace process, running as root.

Setting it to /bin/bash will give you a "Bare and raw" root terminal.  Useful for debug and rescue/recovery.

However.  There is nothing stopping you from setting it to:

/bin/my-gpio-application

and the kernel will honor it like any other userspace "shell" or "init" process.

When that process exits... kernel reboots.  Perfect for most single purpose embedded "applications".

[EVblog1]

What is the purpose of buildroot?

From what I understand, Buildroot is a tool for creating a complete embedded Linux system from source. On the host PC, it builds everything that's needed, including the cross-toolchain, Linux kernel, root filesystem, libraries, user applications, and finally a bootable image. It also provides a menuconfig interface where we can choose which packages and features to include or exclude. Once the build finishes, the generated image (for example, sdcard.img) can be written to an SD card and booted on the target board, such as a Raspberry Pi.

[ejeffrey]

As an asides.  GCC itself will remove pointer access to address 0.  I found that out the hard way when working with the 68k arch on baremetal.  I wanted to test the ASM "bootloader" had correctly mapped RAM to address 0 (ROM on boot).  So I do a write/read on address 0 in a test harness.

It's not just GCC.  A literal "0" is a null pointer in C regardless of the underlying representation.   If your platform requires/allows access to memory at address zero, your compiler needs to provide a specific documented way to do that, you can't just assign 0 to a pointer value.

Almost always, the memory at address zero is in some way special: it might be the IRQ table or it might be the reset vector.  So the easiest and most generic way to work around this is to define a symbol for whatever you want to put there and use the linker script to place it at address zero.  Offsets from that object will work as expected.

[paulca]

You see when I did virtual memory in Uni... it was Windows 98.  The exercise had us literally write a C program which did indeed write to address 0x0.  We then ran 3 of them in different DOS windows to confirm they could each read and write to a "different" address 0 each.  This was circa 2001 though.  Possibly an old once payware C compiler made free, like TurboC or some other shareware thing.

On "NULL Pointer".  That entire concept is abstract and a made up convention.  You will find many such discussions and arguments around the place.

The act of setting a pointer to 0 as a flag has always been a "C smell".  It got transported into Java and has spread to other languages and made somewhat formal as "null" rather than 0.  Many languages, such as Java have a "null" concept.

However.  A lot of people believe this is a mistake.  Many newer langauges simlpy remove the concept of null entirely.  A value will have a value period or it doesn't exist.  Scala, et. al.

The purests would say that the convetion in C and other such languages of "Initialise and the mutate later" is itself a bad practice.  They would state that the two operations should be compounded together and placed where the instance is first required.  NOT in a pre-declared initialised to 0 "Data table" at the top.

[paulca]

What is the purpose of buildroot?

From what I understand, Buildroot is a tool for creating a complete embedded Linux system from source. On the host PC, it builds everything that's needed, including the cross-toolchain, Linux kernel, root filesystem, libraries, user applications, and finally a bootable image. It also provides a menuconfig interface where we can choose which packages and features to include or exclude. Once the build finishes, the generated image (for example, sdcard.img) can be written to an SD card and booted on the target board, such as a Raspberry Pi.

"buildroot" is just an open source project with opinions which tries to make that process "turn key".  Nothing more.

If it is your chosen tool, then their documentation should have guidance on how to insert your own modules, kernel patches and applications to be compiled "for target" during the process.

As I said before, for a small project you can just forgo the whole shell entirely, and have the kernel execute you applications specifically.

On the compilation of your own stuff, I forgot to mention the old trick, if you have the memory and disk space, just "static link" all your binaries.   Literally include the "used parts" of the libraries into the binary itself.  If you can get to "Only static linked binaries" then you don't even need to compile in the dynamic library loader, nor install configure the "ldconfig" gubbins.

EDIT:  Going one step further... if your kernel is booting and executing a single binary as it's init= param.  Then you can even forego a filesystem.  There are HUGE caveats here.  Technically you need a virtual root and the kernel needs it to respond enough to read that binary from.  The middle ground this is usually placed is in the "initrd" which has clever ways to load it self if you look into it.  Consider the kernel only actually "touches" "/" it has zero care for anything else and the only think it will look for on / is your init= file and later dynamically loaded modules.  The advantage of this "/ is a RAM disk" is that a RAM disk is writable, while the backing media could be an EPROM/Flash memory or a readonly SD Card.

[Siwastaja]

On "NULL Pointer".  That entire concept is abstract and a made up convention.

It's just the "invalid value sentinel" concept, very usable all across programming, with some pros and cons. Yes, arbitrary, made up convention, like most things are.

The alternative is storing the validness separately. Magic values (sentinels) have performance benefits: saves the memory of the separate "validness" marker, and saves the few instructions to load that marker from memory into CPU registers. One load for the value (in this case, pointer) itself, then check for validness, then keep using the value - smaller code than with separate validness variable.

The shortcomings are, you lose the possibility to represent that sentinel in the data, and you need to be careful that every participant knows exactly what the sentinel is.

Code: [Select]

struct
{
    int16_t temp_celsius;
    int16_t is_valid;
} temperature;

vs.

Code: [Select]

#define TEMPERATURE_INVALID INT16_MAX
int16_t temperature_celsius = TEMPERATURE_INVALID;

For the NULL pointers, both shortcomings have sometimes realized: most systems use 0 as invalidity sentinel, and most people think this is always the case. Leaves rare cases when the actual NULL pointer in-memory isn't 0, and people get surprised. And the other shortcoming: some systems do place something at memory address 0. For example, STM32H743 places DTCM memory. It's valid start from 0. And there's normally no reset vector or anything like that there; just the user functions right from the start! So a sensible programmer just chooses to waste the first 4 bytes and write their linker script to start at 0x000004, but that happens through trial and error once you find out that GCC fails your init script when it starts to copy from address 0. ST could have chosen any other address, like on most of their other product lines, say, maybe 0x10000000 or anything. No idea why they did this. Maybe just to confuse people, like they do with their IO and DMA mappings.

[paulca]

Java itself ran into that very issue.  Primitives cannot be "null".

So if you have a method, say:

int getPrice() {}

And it turns out that you can't "get" the price for some reason.  Your only option is to "throw".  Which forces the full "Try/catch" work on the caller.

If you did this in Java >1.4(from memory):

Integer getPrice() {}

Then you can return null.  "Integer" is a standard type, a reference, a pointer.  It can be null.

So the caller convention of:

if( myPrice = getPrice() != null ) {
} else {
 cry();
}

Is available.

Speaking of language oddities. 

if( myPrice = getPrice() != null ) {

The output value from an assignment expression differs greatly between them and will hurt you if you do not pay attention.  Some languages the output of an assignment expression is always "True".  In others it is the value actually assigned.  In others it's "Invalid syntax".

Asides:  been spending too much time in the depth of low levels and ASM these days that when a log file in work told me the process exited with 255, I read it at, "You mean -1?"  Assuming it's a 8 bit rc.

[EVblog1]

I've been trying to find a practical case where writing a custom driver is actually necessary.

Right now I have a Raspberry Pi 3, a DS1307 RTC, an AT24C02 EEPROM, and my own PIC-based board. All of them communicate over I2C.

I think  that for the DS1307 and AT24C02, I should simply use the existing Linux drivers because the kernel already knows how to communicate with those devices.

But if I connect my own PIC board to the Raspberry Pi over I2C, then I think that's where I would need to write a driver. My reasoning is that Linux has no knowledge of the protocol implemented by my PIC firmware. it only knows how to talk to the I2C controller, not what commands my PIC understands or what data it provides.

Am I thinking about this correctly? I'd appreciate hearing your opinion if I've misunderstood something.

[gf]

But if I connect my own PIC board to the Raspberry Pi over I2C, then I think that's where I would need to write a driver. My reasoning is that Linux has no knowledge of the protocol implemented by my PIC firmware. it only knows how to talk to the I2C controller, not what commands my PIC understands or what data it provides.

The native I2C driver already supports bidirectional data transfer with any I2C client. Therefore, a specific client driver is optional, as your user-space application can implement the protocol and communicate directly with your PIC device. A dedicated client driver only makes sense if you want to abstract the protocol and expose a specific driver-level API instead of passing the raw protocol through to user-space.

[Kilrah]

But if I connect my own PIC board to the Raspberry Pi over I2C, then I think that's where I would need to write a driver. My reasoning is that Linux has no knowledge of the protocol implemented by my PIC firmware. it only knows how to talk to the I2C controller, not what commands my PIC understands or what data it provides.

Am I thinking about this correctly? I'd appreciate hearing your opinion if I've misunderstood something.

No. The driver only needs to "send data X to address Y" and vice versa, which it already does.
Protocol, logic, what's sent is part of your userspace application.

[ejeffrey]

But if I connect my own PIC board to the Raspberry Pi over I2C, then I think that's where I would need to write a driver. My reasoning is that Linux has no knowledge of the protocol implemented by my PIC firmware. it only knows how to talk to the I2C controller, not what commands my PIC understands or what data it provides.

You can mostly just use the raw i2c drivers.  Here are some reasons you might use a kernel driver:

1) the device is going to be used by another kernel driver layer.  For instance if your device implements a mass storage API and you want to use a filesystem driver and mount it.  Or your device implements a network interface and you want to use the kernel networking stack.

A specific example of this that is easy to miss is GPIO.  A lot of SoC peripherals use GPIOs for various purposes.  For instance an ethernet controller might need to reset an external phy.  The ethernet block in the SoC doesnt implement a dedicated pin for this, rather the system integrator picks an uncommitted gpio and assigns it to this purpose in the device tree.  The ethernet driver then uses the kernel gpio api to reset the phy when needed.

2) the device uses an interface that doesn't have a generic userspace interface.  Thus would be things like PCIe or AXI, but could also be an I2C device that has an interrupt pin.

3) if you want to give non-root users access to use a device in a limited way.  For instance by reading a status but not changing the operating mode.

4) you want to use the device from multiple processes at once.  Using raw i2cget/set from userspace is prone to race conditions and contention.

[Siwastaja]

Java itself ran into that very issue.  Primitives cannot be "null".

So if you have a method, say:

int getPrice() {}

And it turns out that you can't "get" the price for some reason.  Your only option is to "throw".

There is another option - do it like you would in C:
public static final int PRICE_INVALID = -12345678; // some magic value which could never be a valid price.

It's just that maybe for someone writing and reading Java, this would feel like unacceptably horrible hack. In C, it would be natural, and one would "feel" how it maps to hardware.

Then again, forcing exceptions everywhere is also horrible. Exceptions, after all, are goto on steroids. Exactly the anti-pattern people warn against. Useful, but dangerous, and not trivial by any means. Java programs are well known for "crashing all the time with endless cryptic error messages". That is not fault of Java itself, but caused by programmers being unfamiliar with exceptions and suddenly having to use them, in late 1990's / early 2000's. (At least in C++, not many standard library functions throw exceptions on every little thing, and old C-style error checking is a viable alternative pattern.)

[nctnico]

Java itself ran into that very issue.  Primitives cannot be "null".

So if you have a method, say:

int getPrice() {}

And it turns out that you can't "get" the price for some reason.  Your only option is to "throw".

There is another option - do it like you would in C:
public static final int PRICE_INVALID = -12345678; // some magic value which could never be a valid price.

It's just that maybe for someone writing and reading Java, this would feel like unacceptably horrible hack. In C, it would be natural, and one would "feel" how it maps to hardware.

Not everybody thinks that way. MISRA coding rules explicitely forbids using sentinel values.

[Siwastaja]

Java itself ran into that very issue.  Primitives cannot be "null".

So if you have a method, say:

int getPrice() {}

And it turns out that you can't "get" the price for some reason.  Your only option is to "throw".

There is another option - do it like you would in C:
public static final int PRICE_INVALID = -12345678; // some magic value which could never be a valid price.

It's just that maybe for someone writing and reading Java, this would feel like unacceptably horrible hack. In C, it would be natural, and one would "feel" how it maps to hardware.

Not everybody thinks that way. MISRA coding rules explicitely forbids using sentinel values.

You know, this isn't true at all. Such MISRA rule does not exist. Like, people can actually fact-check what you say. Feel free to show where sentinels are explicitly banned. The whole standard doesn't even mention the word sentinel.

Some types of sentinel (mis)uses are banned through other rules, like comparing -1 against unsigned operand.

Magic numbers are fine per MISRA rules. But they of course need to be clearly named constants (e.g. #defines), not just number literals littered around code (readability rules).

[SiliconWizard]

I'm not aware of MISRA-C forbidding the use of "sentinel" values. All it forbids is the use of hard-coded values, so indeed you would clearly define your sentinel values instead of using them hard-coded in your code.

There may be a confusion coming from this specific rule: MISRA C:2023 Rule 11.9

The macro NULL shall be the only permitted form of integer null pointer constant.

It specifically applies to the use of an integer null pointer constant, not just any sentinel value in your code. We can say that NULL is a sentinel value for pointers, but that doesn't generalize to anything else.

Now, this rule implies that you should not use any other sentinel value for a "null pointer constant" and not even use another macro definition than NULL.
This is a specific case.

[nctnico]

I'm not aware of MISRA-C forbidding the use of "sentinel" values. All it forbids is the use of hard-coded values, so indeed you would clearly define your sentinel values instead of using them hard-coded in your code.

I think that is it indeed. It came up in a code review a while ago.