From power on to login prompt, how does Linux boot, step by step?

[RoGeorge]

Most probably I will be confusing dedicated software terms and wrongly use them as generic words, so bare with me please:

Looking to clarify the main steps for the Linux booting process
- a bird eye view at first, with all the key concepts and steps
- some details of each step and each method (by "method" thinking here SysV, OpenRC, Systemd, etc. - no idea how/what each is/do)
- how all pieces click together, and come up alive, so to later start interacting/talking together in such a way that in the end I could run the same program without bothering what boot method was used
- very interested in very first steps, close to hardware

For example, what would I want to know if I want to solder together an ARM and some RAM on a board (or maybe just use an FPGA devboard at first), and make them boot my own GNU/Linux? No LAN at first and no fancy multimedia. Just a board with a serial port where to connect an external TTY terminal.

Things like that. Did searched online many times before about the booting process (also not very sure what part of the whole process between powering up and login prompt is called "the boot process" in Linux), but only found either way too generic explanations with no examples, or too detailed things at the level of very specific options/switches.



TL;DR
Any good resource/link/advice to understand the boot process, with the goal of putting Linux on a new type of a DIY board (ARM based, or maybe MicroBlaze)?

[Nominal Animal]

Hopefully someone else can provide links.  All I can really think of is to recommend looking at openwrt, especially mounting some of the images and exploring their contents; and at LinuxFromScratch for detailed information on how to build an entire Linux system from sources (but note that it is targeted towards general purpose Linux installations as opposed to embedded systems).

Here's a quick overview of the entire boot sequence.

• Power is applied, and the hardware begins the boot-up sequence, by running from internal ROM to do self checks (BIOS, EFI) and to load a boot loader.  Multiprocessor systems typically only bring up one processor.
• The bootloader is loaded into memory.  The two most commonly used are U-Boot and Grub.
• The bootloader loads the kernel into memory, and gives it control of the system.
• The Linux kernel loads and mounts a ramdisk, and executes init (/sbin/init) as process 1 from it.
  
  This point is where embedded systems and appliances diverge from general purpose systems.  For general purpose Linux installations, the ramdisk is just an initial ramdisk containing kernel modules (at least those needed early in the boot process) and an early boot init script, which brings up the rest of the system; in particular, mounts the system partitions, then pivots to the new root partition, continuing the (main) boot process there.  On embedded systems and appliances, the ramdisk contains the entire system as-is.

The init process always has process ID 1, and has special responsibilities:

• init cannot and must not exit.  If it does exit or crash, the system will panic (crash and become unoperable).  This also means it has to handle POSIX signals in a sensible manner.
• Processes whose parent has already exited are orphans, and re-parented to init.  Because zombie (dead but not reaped) processes have some kernel resources attached (PID, exit status), init must reap any children (preferably without delay).
• init is responsible for maintaining console terminals (getty) for user login.  When an user logs in, the getty is replaced by /sbin/login, which forks and executes the shell for that user.  When /sbin/login exits, init must start a new getty for that console terminal.
• init is responsible for starting the rest of the userspace, by forking a child process that starts executing the boot scripts.

Because there are several different ways to achieve the above, and init is not just a single binary but also dictates the ways above is achieved, we call the implementations init systems.  For example, Mac OS uses launchd.  SysV-style inits are the simplest, as they use shell scripts and a few helper programs (runlevel, telinit), with a specific directory hierarchy and command-line interfaces; they are much more about where things are, than what does what.

A minimal system only requires a trivial (or SysV-style) init process, and a shell (directly started by the init process, connected to /dev/tty0).  BusyBox is often used to provide most of the utilities an user might need running such a shell, but full-sized Linux systems have Coreutils.

If you have a desktop or laptop Linux machine capable of running (small) virtual machines, I recommend installing a truly minimal Devuan to a virtual machine, and exploring its internals, including the initial ramdisk (/boot/initrd* cpio archive).  Devuan is a Debian derivative which untangles the pervasive tentacles of systemd, and actually allows you to install a system using other init systems.  After installing, checkpoint the virtual machine, so you can revert to that state (including disk contents) after you make it unbootable.  (If you do not make it unbootable, you're not playing with it hard enough.)  A particularly good exercise is to use sysvinit, rip out all boot scripts, and use a single Bash script to do something, say with bash-builtin printf with nice colorful ANSI escape sequences, say
    printf '\033[0;32m[\033[1;32mOk\033[0;32m]\033[0m\n'
for [Ok].

Just do not make the mistake of comparing the simplicity and robustness of such systems to that of currently popular systemd-based ones, or you will find out why so many of people like me really, really dislike systemd; that hatred truly has nothing to do with disliking change or not liking new stuff.

[dgtl]

Take a look at the http://www.linuxfromscratch.org/ free book, that walks through the build process of a linux system. This gives a good insight, how the thing is done.

To add to previous post:
In embedded systems, the system may also be without ramdisk - the root partition is directly mounted and init started from there. A ramdisk is just a small filesystem image, that is put to memory by the bootloader and the system can start executing stuff from there before it has access to the actual storage hardware (ie first drivers are installed from ramdisk to kernel, partitions get autodetected, nework gets started to mount network file systems etc). A system without ramdisk must have drivers to access the root file system and the whole logic of getting things running built into the kernel itself.
So you have here 3 options: no ramdisk, root filesystem only; ramdisk-only, system runs from ram; or ramdisk and then pivot to root filesystem later.

Also, in ARM systems, the bootloader loads a devicetree file to memory and provides the pointer to kernel. The devicetree is like a configuration file to kernel, because large parts of ARM systems usually do not support plug-and-play autodetection stuff for most of the peripherals (excluding USB, PCI stuff of course). The devicetree tells, what devices are at what addresses/irqs for all the memory-mapped stuff (ie UARTs), but also for external stuff like I2C and SPI - connected things. So for ARM systems, you shall have both the driver and the devicetree record to show what driver to use where.

In short - as the bootloader must be capable of reading some kind of boot media to load the stuff, it usually loads kernel, ramdisk and/or devicetree to memory and then executes the kernel.

[retiredfeline]

Joke: How can you tell the systemd haters?

A: You don't, they'll tell you. 

[nightfire]

Not exactly Penguin-style, but a look to another Unix System could also prove some insights how things can be achieved:
https://wiki.freebsd.org/arm/BootProcess

*BSD does some things different from Linux, but the documentation usually is quite good and has sometimes a very different angle of view from the Linux Way of Life- usually worth a look.

[DC1MC]

If it's an embedded ARM system I have experience with some Cortex A  MCUs, like NXP i.MX and sunxi series (A10, A20 and so on).

So basically almost nobody does "linux from scratch" on these embedded systems, most of the manufacturers offers Board Support Packages (BSP) that will build a firmware for their development kits and maybe some of their partner OEM boards. These kits comes mostly in source (except the graphical crap) and should be used as templates for your design.

The collection of spaghetti scripts that builds your firmware are called "buildsystems" and I mention the most notorious and miserable Yocto, followed by PTXDist, Buildroot and Armbian, all of them support a crapload of boards, came with reasonable documentation and have support forums and mailing lists, usually pretty helpful. Normally nobody build a configuration from absolute scratch, because it's almost impossible to have a MCU that runs Linux and not having it covered by one buildsystem or another, usually by more. So is just a matter of getting an existing configuration and tweaking it to your needs.

On the highest level, usually the MCUs have a bit of masked ROM ( usually called BootROM), some miserable amount of static RAM (in range of tens of KB) and this is what you have to work with to boot your system.
At power on, the control goes to the BootROM, this usually reads some external boot pins or internal OTP registers, that select the boot media (SPI Flash, NAND FLash, SD/MMC, USB and so on), the boot media has to have the first sector containing the a special formatted structure that describes how the boot loader (usually U-Boot, some times BareBOX or some proprietary contraption) should be loaded, the bootloader main task is to prepare the DDRx memory, by configuring the the memory controller and running calibration tests and then load the device tree (hardware description) and the Linux kernel.

Once loaded, the kernel is feed with the RAM size and the device tree and then booted, from there on, Linux will figure how to proceed

[free_electron]

There is a BIG step missing in all these descriptions : the Memory initialisation ! Without that NOTHING will happen. And that is NOT part of the OS !
When the processor powers up and comes out of reset it has no idea what memory is attached. and we are not talking size of the memory. We are talking the timing  , refresh , burst sizes etc. There is a minimalistic chunk of code. sometime sonly a few tens to hundreds of bytes long. This code sets up the memory controller with the correct information for timing but also creates the memory map. the ram sets here, the flash there etc. This may include the flash memory ! Upon cold start the flash controller defaults to a specific address with fixed parameters. This code need to reside there. Once this work is done the memory controller is instructed to 'remap' the flash. It swaps out this piece of code with the real OS image and executes a soft restart of the processor. The difference between a soft and hard restart is that the memory controller does not go to power on reset state. it retains whatever as set-up by the bootstrap code.

When i was working on the Arm7TDMI this boot code resided in the top page of the flash ( top 512 bytes ) . By default , after a cold start, the processor would jump there and start executing. The processor accesses 0x0000_0000 but in reality this is mapped to the top page
) There is no 'copying to ram' or anything. It runs from rom as the dram controller is not set up. There were a few MOV operations that basically set hardcoded numbers into specific memory locations tied to the dram controller. Then the dram controller started up and ram became available. The last thing was to instruct the flash controller what offset to use . After cold start the offset was 0xFFFFF800 , pointing to the top 512 bytes of the memory range. if you had a 2 meg flash you would now set the offset to (FFFF_FFFF - 2 meg) so that becomes '0'. and then a soft restart was executed. The system would now start with the new 'base address' and execute whatever code is there. That would be the bootstrap for the OS.

How to set up the memory controller depends on what ram and flash you use. consult the datasheets. Most modern systems support a few rams and flash chips out of the box and the processor manufacturer will give you the few bytes needed to do the start. all you have to do is add these to the rom image at the right location. Some processors have a dedicated config flash or eeprom to store this in and come with a preloaded config for some common ram and flash. then it is a matter of just using that type of flash and ram and wire it as given. if you want to deviate or alter the layout of the memory you will need to alter the BSP. ( board support package : the bootstrap code that initializes the memory controller so that a generic bootloadr can now start) . the BSP may do other things too : temporarily disable the watchdog for example until the OS can do its thing. It may also configure certain I/O pins or read config straps. Some processors have a 'sealing' operation. Once the config is set the access is 'sealed'. The settings cannot be changed until the next hard reset. This prevents runaway programs from altering critical settings in the memory timing controller.

[DC1MC]

And to continue on the miserable task of setting up the DDRx, this is indeed the largest and most complex task of the bootloader, the higher you get with x, the most miserable is.

For example on the i.MX series is like this:

DDR2 - yeah well, with a bit of pain one can calculate the register values and even borrow the values from another project and this is it.

DDR3 - There you need a manufacturer spreadsheet (yes, Excel with macros) where you input your chosen RAM model parameters AND afterwards, with the initial values of the RAM block registers one needs to run a "training" special program on the actual hardware, that will iterate with this pre-calculated values and try to find some suitable for your board delays. Of course over USB, because why not, welcome "production tools".
Bonus 1: if one has a PMIC it needs to start it with register level commands, as well as the clock tree.
Bonus 2: one has to run this "training" program on a wide range of temperatures to get some stable values.
At least the values of the memory controller are fixed and once you get them for a hw version, you're free to go and proceed with the production.

DDR4: Here usually you have a MCU in the MCU (or more, i.MX8 has two, a MCU and specific firmware for the DATA memory and for the INSTRUCTION memory RAM controllers). Then the memory is totally unstable at any temperature, at every startup a calibration routine MUST be run by the bootloader, if the calibration pass, the memory is initialized and workable, if not, you're in a world of pain...

DDR5: I've never used one of these, but I don't believe it becomes simpler, time will tell.

And these is for the discrete, well supported MCUs, for the FPGA variants, take all of the above and multiply it with at least four times.

Yeah, making the RAM available to Linux is a pain on modern MCUs 

[free_electron]

DC1MC has a valid point too. if you are going to use fast memories the board layout becomes a problem. Skew in the data and address path becomes a problem. The controller needs to be 'trained' so it can do some measurements and use that set of data as initialisation. Touch the PCB layout on the memory side and get ready to run that stuff again. It's all fun and games up to 200MHz. beyond that getting the memory to run properly becomes an issue.

[Nominal Animal]

In embedded systems, the system may also be without ramdisk - the root partition is directly mounted and init started from there.

Yes, but I do believe it is more common in older desktop systems, actually.  It requires that the partition is not compressed, which makes it problematic on non-SSD Flash-based media with limited write cycles (especially SD cards), as the longevity of Flash can be severely negatively impacted unless it is mounted read-only.  Mounting the filesystem read-only means additional integration work, since many services (and even core OS components like mount) really don't expect that, and need workarounds. Plus, you need relatively lots of Flash for it.  It is more common to have a smaller Flash but more RAM, with Flash containing a compressed image of the filesystem, making it inherently read-only; and since the filesystem is in RAM, it does not need to be read-only, making the userspace easier to integrate.

In some cases, like Live installations, you can combine the two using filesystem overlays.  Basically, your base filesystem is mounted read-only, but is overlaid by a RAM-based filesystem overlay that contains the differences to the read-only data in RAM.  These can be very useful in many cases, especially since you can easily detect modified files, and copy them to semi-permanent storage.  The downside is that long-running overlays collect also unneeded data, especially filesystem metadata (access timestamps), so the memory needed by the overlay tends to slowly grow, making it difficult to estimate the long-term RAM requirements of such systems.

There is a BIG step missing in all these descriptions : the Memory initialisation ! Without that NOTHING will happen. And that is NOT part of the OS !

Very true.  It is the part that comes before even early boot, before even the bootloader, and is part of the stuff one needs ROM for; the part equivalent to BIOS and EFI on desktop machines.

I glossed over it, because I've personally never implemented it myself; only stuff from the bootloader onwards.  I'm glad you and DC1MC brought it up.

Joke: How can you tell the systemd haters?

If you knew what it really is and what it does to your system, you'd hate it too.  There is a reason it is never used in embedded systems.

About a decade ago, the Linux kernel live-patching system became so robust, that you could fix a running kernel without having to reboot.  Then we got kexec, which allows a kernel to essentially restart or switch to a newer version without having to reboot.  We had robust, stable machines with high uptimes, and reboots were rare; we could even create and apply security patches without rebooting.  Powerful times.

Nowadays, because of systemd, Linux distributions require regular reboots because of userspace updates.  Postpone them, and your system starts behaving oddly.  Some, like Debian, will most likely switch to a two-reboot sequence to do core userspace updates, simply because the structure and implementation of systemd cannot handle severe changes in the running system.

We have gone backwards, several decades, and completely lost the gains made in software engineering in the last two decades.  Why?

You really, really don't understand the harm it has done until you investigate systems integration at the lowest level, like RoGeorge is going to do.  And when you do, you either accept that crap tastes acceptable because billions of flies cannot be wrong, or you become a "systemd hater".

You joke, but it is a deadly serious matter if you care about your tools.  It is like stainless tool steel being replaced with low-grade cast aluminium, because it looks sufficiently similar, but is easier to produce and polish; and most people need neither the longevity or the strength of stainless tool steel.  And you are the one making jokes about those complaining about the replacement.

Shame on you, retiredfeline.

[Ed.Kloonk]

My daily driver distro does not use systemd and it's lead developer vows it never will.

One of my biggest gripes about *buntu is the blank screen during startup. I know you can turn it off and I know you can hit esc during a boot, but the point is that boot up text can often reveal failing hardware and to hide that with a blank screen or a dumb animation is offensive.

[retiredfeline]

If you knew what it really is and what it does to your system, you'd hate it too.

My system has been systemd based for years and I have years of experience managing enterprise systems so I actually know what's happening. In short, meh, suit yourself, it doesn't happen to my home system, and didn't happen to the systems I used to manage. I also managed SysV (and before that BSD RC, and the other boot systems all the way back to PDP-11 Unix) systems and I have no issues with that either. I tamed.both and it just.works.for.me.

I have no issue with people who have technical or aesthetic objections to systemd. However for many others it's an identity thing and their arguments have errors (hint, it's not a boot speed thing) their constant whining ruins forums. I just laugh it off. 

For embedded systems I have no opinion since I'm not interested in Linux for this space at the moment.

[Nominal Animal]

If you knew what it really is and what it does to your system, you'd hate it too.

My system has been systemd based for years and I have years of experience managing enterprise systems so I actually know what's happening.

It's not bad from the management perspective, sure; it's got polish and nice user interfaces.  That's why so many end users and admins either or at those who complain about systemd.  I do not believe you actually know, truly, what is happening, because you are just an experienced user and an experienced system administrator without systems integration experience.  You only see the polish, not the sickly innards.

You probably also care much less about uptime than you used to, because most important systems tend to be virtualized or farmed, so rebooting one is not an issue.

My own background in system administration is in web services and HPC (large clusters doing relatively long-running MD simulations), but I've done a fair bit of systems integration work – including dedicated "mini-distros" for very specific purposes –, and a fair bit of OS development and Linux kernel work.  Not really a domain expert, though; more a wide-ranging generalist, really.

I tamed.both and it just.works.for.me.

That's my gripe.  You say you tamed it, but what you actually did, is just use it out-of-the box, and are happy with its polish and function for your use cases.  You care nothing about how it affects the underlying stability, configurability, or modularity of the system, as long as it does not bite you.  It works, nobody yells at you, you get paid, you're happy, so everyone else is just being stupid.

Most of the paid systemd developers work for companies whose target "demographic" includes you at the center.  (Technically, they target companies, and those companies hire sysadmins like you, but meh.)  It would be very bad business if you weren't happy.  Canonical (Ubuntu folks) also spend quite a bit of money to keep the polish for single-user desktop.   Nevertheless, neither is any kind of reliable indicator of the quality of the product, just that it is well targeted at and polished for its niches.  This I well understand; most users are quite happy with it, because they don't need to deal with any of its weaknesses and uglinesses.

I have no issue with people who have technical or aesthetic objections to systemd. However for many others it's an identity thing and their arguments have errors (hint, it's not a boot speed thing) their constant whining ruins forums. I just laugh it off. 

What you mean, is you don't care if there are technical objections, because you feel they don't affect you, so those complaints are just annoying forum-ruining whining.

Yeah, right.

[retiredfeline]

You probably also care much less about uptime than you used to, because most important systems tend to be virtualized or farmed, so rebooting one is not an issue.

You are just shooting in the dark and very mistaken. These were desktops and servers used by engineers working at a company that paid 6 figure license fees per year for CAD software, that stayed up until some other situation required a reboot, like the engineer moving desks. In fact I had to look for opportunities to reboot to install fixes for Spectre and other kernel bugs. When it came to the crunch, kernel bugs had to take a back seat if the host had to stay up because the engineers as usual had left the desktop where they left off, or were running a long running simulation. There were also virtualised servers and those had the same sorts of uptimes. We also had no support contract. We just used the public repos. I installed a CMS (Puppet) to manage the infrastructure, and SSO integration with Windows servers, so I'm not a colouring book kind of sysadmin either.

Maybe your experience is different. IDGAF. I earned a good living and I laugh, well nobody goes to the bank any more, so everytime I pay by tap. 

[Nominal Animal]

Your popular mass adoption distributions like Fedora, Ubuntu, their vast derivatives, and the few commercial distros, all use it. I wonder why.

Because Fedora and Ubuntu pay the systemd dev salaries?

The Debian CTTE mailing list is an interesting place to look for conflicts regarding this stuff.  The Debian systemd maintainers are blocking all requests that would allow users to switch to any other init system, or to run without systemd installed, and the conflicts are often escalated to the CTTE.  The Debian systemd maintainers are even actively dropping stuff in systemd-subsumed non-init packages that others rely on to run non-systemd inits, and using socially aware language ("we already agreed to this", "it is offensive to...") instead of technical arguments for doing so.  Insiduous, isn't it.

Basically, they're making sure nothing competes with systemd from now on.  They've already done that for Fedora and Ubuntu derivatives, and are fighting hard to ensure it happens on Debian also.

[DiTBho]

For example, what would I want to know if I want to solder together an ARM and some RAM on a board (or maybe just use an FPGA devboard at first), and make them boot my own GNU/Linux? No LAN at first and no fancy multimedia. Just a board with a serial port where to connect an external TTY terminal.

what you have to care about is the first process to load.
Usually it's /sbin/init, but you can force /bin/bash.