Monolithic or microkernels?

[SiliconWizard]

Yet another debate. Not exactly meant to revive the oldish debate between Linus Torvalds and Andrew Tanenbaum (author of MINIX), although... why not.

Do you think Linus was right and microkernels make no sense from a performance POV, and the other alleged benefits are not enough to counterbalance it?

[DiTBho]

Do you think Linus was right and microkernels make no sense from a performance POV, and the other alleged benefits are not enough to counterbalance it?

I think Linus was right when he said it because microkernels consume more resources and when Linux took its first baby step (v0.0.1) ... we had i386SX25 with 8 Mbyte of ram

microkernels have always been superior on paper... but they weren't that great in practice... and if you look at thier legacy, well ... "Haiku" is based on NewOS Kernel, but it's a hybrid solution 

[eutectique]

I think Linus was right when he said it because microkernels consume more resources and when Linux took its first baby step (v0.0.1) ... we had i386SX25 with 8 Mbyte of ram

At the same time:

• OS-9 2.x was running on M68000 with 1 MB of RAM and 2 MB of flash
• QNX 2.x was running on i286 with 2 MB of RAM
• QNX 4.x was running on i386EX with 2 MB of RAM and 4 MB of flash

[mapleLC]

I think Linus was right when he said it because microkernels consume more resources and when Linux took its first baby step (v0.0.1) ... we had i386SX25 with 8 Mbyte of ram

At the same time:

• OS-9 2.x was running on M68000 with 1 MB of RAM and 2 MB of flash
• QNX 2.x was running on i286 with 2 MB of RAM
• QNX 4.x was running on i386EX with 2 MB of RAM and 4 MB of flash

Isn't containerized virtualization basically a useful implementation of this?

When these things are said, they can be in jest, but otherwise a limitation in the person's thinking at the time.  Bill Gates numerous times, but most famously saying that 640K of RAM ought to be enough for anybody.

I think the argument is settled, the microkernel has the utility it lacked when Linus made those comments.

[chickenHeadKnob]

I think Linus was right when he said it because microkernels consume more resources and when Linux took its first baby step (v0.0.1) ... we had i386SX25 with 8 Mbyte of ram

At the same time:

• OS-9 2.x was running on M68000 with 1 MB of RAM and 2 MB of flash
• QNX 2.x was running on i286 with 2 MB of RAM
• QNX 4.x was running on i386EX with 2 MB of RAM and 4 MB of flash

My experience with QNX in the 80's, version 1.x something and higher settled my mind on the performance issue. It ran fine on something as wimpy as 5MHz 8088 pc. I laughed at LInus's derisive they are just "message flingers" . Exactly, that is what is so good about them. For a modest performance hit you get great process isolation and network operation.

[alm]

I think the argument is settled, the microkernel has the utility it lacked when Linus made those comments.

Which practical application do you have in mind? Or are we talking about it being settled on paper? All current applications I can think of settled on hybrid approaches: Mac OS X used the Mach micro kernel, but then only has only one Mach service, so there's very little difference with a monolith. Windows NT started off as a hybrid approach, but since moving the graphic subsystem in kernel mode with NT4 is barely different from a monolith.

I think Linus was right when he said it because microkernels consume more resources and when Linux took its first baby step (v0.0.1) ... we had i386SX25 with 8 Mbyte of ram

At the same time:

• OS-9 2.x was running on M68000 with 1 MB of RAM and 2 MB of flash
• QNX 2.x was running on i286 with 2 MB of RAM
• QNX 4.x was running on i386EX with 2 MB of RAM and 4 MB of flash

Ancient operating systems run on ancient hardware (QNX 2.0 was from 1987, and I imagine OS-9 2.x was even older since the first release was in 1979). Especially operating systems targeting embedded platforms. I don't see what that proves. There were also versions of (monolithic) *nix that ran on the 8088.

[SiliconWizard]

One obvious issue with this "debate" is that one should clearly define what kind of "microkernel" one is talking about when making any kind of comparison.
There's a very general architecture, but beyond that it's very much down to each individual implementation.
Even MINIX is not quite what it initially was, and MINIX3 seems not to be too shabby.

While some microkernel implementations have "failed" in the sense of not getting really anywhere, QNX is clearly a successful implementation of the concept.
XNU, the macOS kernel, which is more identified as an "hybrid" between microkernel and monolithic, is a clear example that it also works, even for desktop use. To which extent its hybrid nature has helped it be a workable option rather than if it were a 100% microkernel, I admit I have no clue.

seL4 is also a current example of something that seems to work just fine.
https://sel4.systems/

[Marco]

Do you think Linus was right

No, message passing with pass by reference can be made fast enough to connect kernel components. See L4.

Address space separation within the operating system at the time was not an option, but that's not what defines a microkernel. Nowadays with PCID it's an option.

PS. device mapper is a good example of just how much function call indirection is doable for essential OS services. LVM based storage is already a giant pile of indirection, fast path message passing wouldn't break the bank.
It's not like microkernels make every function call a message, sane OS components don't have API call frequencies high enough to matter.

[Nominal Animal]

Do you think Linus was right and microkernels make no sense from a performance POV, and the other alleged benefits are not enough to counterbalance it?

Then, yes.  Now?  I can argue both ways, so it highly depends on the details like purpose, maintenance and development process, license, and programming language.  And target hardware, of course.

For heterogenous multiprocessing (different cores or distributed processors), microkernels fit much better than monolithic kernels.  We can see this in various computing sub-fields from phones and appliances (with dedicated cores) to distributed computing in HPC to server virtualization.

Really, the hardware has progressed much further than the programming languages we humans use to control the hardware.  To me, the culprit is not actually the programming languages themselves, but the limitations in human understanding and control of complex systems.  Synchronous vs. asynchronous is a prime example of this.  Most programmers, regardless of language or toolkit or development environment, still have issues in designing robust and correctly operating asynchronous systems.  Add in parallelism, and the set of programmers capable of fully understanding the system and avoiding the typical pitfalls (deadlocks, unnecessary exclusion via locks, "global locks") shrinks to a surprisingly small set.  To those who haven't done it themselves, and to those who do not understand that true development is a highly creative process, this seems to be near impossible to accept: they seem to believe any computer problem (design or not) to be solvable by simply throwing more mediocre programmers at it, until something sticks to the wall.

The kind of design architecture I believe future general purpose computers will need, is the kind where you can online and offline individual heterogenous processors and update kernel software in run time, without "rebooting" the system.  Something like cooperating microkernels, with no dedicated "boot processor", perhaps.  This belief is based on both the technological capabilities (inherent physical limitations in the capability of a single-core processors), and the projected/assumed/extrapolated needs of the human users, especially energy conservation, and high-bandwidth data flows.

[SiliconWizard]

Three major points that I see in favor of the "microkernel" approach - which I would actually like to see in software in general, not just kernels - are: 1/ isolation, 2/ modular designs with small, manageable chunks and 3/ message passing and much saner handling of resource sharing.

Now of course, what may have been problematic 30 years ago on 386 computers has become irrelevant to a significant extent.

[DiTBho]

• isolation
• modular designs with small, manageable chunks
• message passing and much saner handling of resource sharing

well summarized 

Now of course, what may have been problematic 30 years ago on 386 computers has become irrelevant to a significant extent.

Thinking about my IDT-R2K board (MIPS), comparing with my Atlas MIPS5++ prototype
- with MIPS-2K (as well as with m68k) there is only one ring of isolation, { superuser, user }
- with MIPS5++ and ARM there are more rings, and += hypervisor
- the MIPS5++ introduces the precious tr-mem, which simplifies and accelerate the message passing mechanisms

previously microKernels pushed the kernel into the "superuser" ring, all other services { network, filesystem, ... } in the "user" ring, and user applications in the same "user" basket, with some #trap to let them communicate.

#traps, especially on m68k (from 68000 to 68060), have a lot of overhead. MIPS has less, but trmem can help here, a lot.

Dunno about i386 

[Nominal Animal]

Three major points that I see in favor of the "microkernel" approach - which I would actually like to see in software in general, not just kernels - are: 1/ isolation, 2/ modular designs with small, manageable chunks and 3/ message passing and much saner handling of resource sharing.

Now of course, what may have been problematic 30 years ago on 386 computers has become irrelevant to a significant extent.

Quite.

As an example of the technical issues with isolation, consider a set of three processes A, B, C that circulate data.  A is only allowed to see what C sends, B is only allowed to see what A sends, and C is only allowed to see what B sends, so the three form a unidirectional cyclic graph.

You can implement this with three shared memory segments (A-B, B-C, C-A), with two of them mapped in each process.  Processing data involves copying the data from one segment to another, then signaling the next process that the chunk is ready.  Simple and robust, but copies data around.

Or, you can implement this using message passing with the data as one or more RAM page payload, with the kernel modifying the page virtual memory attributes.  This is the zero-copy approach, but involves quite a few caching-related details.  It is by no means guaranteed that this is faster than the copy approach, especially because this involves crossing the userspace-kernel privilege boundary (although so will cross-process locking primitives).  A particular corner case is B receiving a chunk from A, making a small modification, and forwarding it to C.  To be secure, A must not see the changes B makes to the data, and this can be difficult to ensure if A and B run on different cores.  (I don't even want to speculate on the issues if such transferred chunks contain executable code, and the interaction with speculative execution.)

It is the caching and cross-core synchronization wrt. virtual memory that makes the most theoretically interesting options not so simple in practice.  If performance is the key factor considered, then it is not at all obvious which one (monolithic or modular microkernels) is superior; it really depends on the hardware.

Isolation itself is an interesting subject, because while I like the old-style Unix user-group model, I could really use "sub-users" for process isolation and access control.  For example, if I start a process A, it would be very useful if A could start an isolated agent B that cannot directly access my private data, only data accessible to my own group; requiring A's authorization (on my behalf) for accessing my private data.   Similarly, it is not necessary for all kernel subsystems to be fully isolated from each other.  Permission tokens might be a way to implement something like this (as well as current models) more sanely, while giving robust security guarantees.

[SiliconWizard]

If you haven't, you could have a look at the MINIX(3) project. I'm not saying to actually use it, and have no opinion on the implementation, but many of the concepts it holds are exactly that.

[Nominal Animal]

If you haven't, you could have a look at the MINIX(3) project. I'm not saying to actually use it, and have no opinion on the implementation, but many of the concepts it holds are exactly that.

Sure.  For those who want to follow up on that, look at MINIX message passing (fixed-length messages with 56 bytes of payload) and memory grants AKA safecopies.

MINIX 3 is also POSIX conformant, meaning that even though it is a microkernel, software written in C to be POSIX-portable (say, across Linux/Mac/BSDs), should compile and run on MINIX 3 just fine.

[SiliconWizard]

I didn't know a whole lot about the details of the "argument" between Linus and Andrew (MINIX) until recently.

Apparently, the relatively strong reaction from Linus was not just about technical facts - it was, of course, but it's easy to understand it when you see that Andrew had been pretty bold by claiming in a famous 1992 note that "Linux was obsolete". Yeah, he started it!

Andrew was more generally claiming, back then, that monolithic kernels were obsolete and that the future would be microkernels.
At the time, he even cited Windows NT as a microkernel, although at the time it was merely speculation. Windows NT wasn't out yet. But it is known to be based, at least to a large extent in terms of architecture, on the MACH kernel. In the end, Windows NT is definitely not a microkernel, but it can be seen more like a hybrid kernel - like XNU for macOS - and in that regard, while his claim was pretty bold, even to this day, Linux is one of the few widely-used, general-purpose kernels out there that is purely monolithic.

[RoGeorge]

Well, if we count which one runs more copies today, decades later, I'm inclined to believe MINIX has more running instances than Linux, as we speak.  This is because any current Intel processor has a MINIX kernel inside. 
https://www.cs.vu.nl/~ast/intel/

[SiliconWizard]

Yes, IME is apparently using MINIX. Its author didn't particularly appreciate that Intel never cared to inform him about it.

[DiTBho]

I don't have to talk too badly, my file-system is much worse, but ... the Minix file-system, even if better than ext-v1,  is too catastrophe prone, and still needs to be improved: never compile it with large file caching!

(running Minix-v1 on my i486 laptop
it boots from a PCMCIA flash card)

[Marco]

It is the caching and cross-core synchronization wrt. virtual memory that makes the most theoretically interesting options not so simple in practice.

In your example, the kernels both have the exact same choices, the kernel copies the data or it remaps shared memory with all the TLB shenanigans involved (hopefully the architecture is sane enough the data cache can be left alone, with only the TLB cache being involved). The overhead of message passing is defacto irrelevant and they have to do the same things on the hardware, so it really doesn't matter.

[Nominal Animal]

It is the caching and cross-core synchronization wrt. virtual memory that makes the most theoretically interesting options not so simple in practice.

In your example, the kernels both have the exact same choices, the kernel copies the data or it remaps shared memory with all the TLB shenanigans involved (hopefully the architecture is sane enough the data cache can be left alone, with only the TLB cache being involved). The overhead of message passing is defacto irrelevant and they have to do the same things on the hardware, so it really doesn't matter.

I did not intend the preceding paragraphs to be an example of caching and cross-core synchronization, but as an example of issues with isolation.
I probably should have put a dotted horizontal line or something to indicate focus change.

(When you consider a message-passing-based microkernel and a monolithic kernel, yes, they both can utilize the same methods.  However, whether you can rely on all hardware being able to pass a reference across privilege boundaries (between different processes) and maintain privilege separation correctly, i.e. isolate processes reliably when doing large data transfers that way, is in question.   This means that while in theory, one could claim that say "message passing has less overhead, because it allows passing only the reference to the data, without copying the actual data", it is not generally/universally true in practice.  Marco is also right in that that statement would be silly/malformed/wrong, because monolithic kernels can do the same.  Switching the point of view to isolation, you see my point: isolation is much more complicated in practice than it sounds in theory.  It is also why I believe security has to be designed in to the very core, and cannot be added on top later on.)
 _ _ _

Note that overall in this thread, my focal point is that the practical details like hardware architecture and even programming language used, affects my opinion on which I would prefer, monolithic kernel or microkernel, even if using pure performance as my only measurement.

Currently, the hardest topics to 'grok' seem to be the interaction with privilege separation and caching mechanisms.  We see this in the relatively recent design bugs wrt. speculative execution.  If you follow e.g. the Linux kernel mailing list, caching-related bugs and hardware fixes tend to occur at a constant rate that only depends on the complexity of the caching architecture; with typical problems stemming from the fact that humans did not understand all the implications of the designed behaviour, rather than silicon bugs.
 _ _ _

If we harken back to the argument between Torvalds and Tanenbaum, one way to look at it (if we are neither of them ourselves) is to see it as former arguing based on practice, and the latter arguing based on theory and models.  Neither was wrong, per se, in my opinion, exactly because theory ≠ practice; and the argument itself was only possible between humans that place very different importance or weights to practical implementation vs. theoretical basis.

[Marco]

With a programming which guarantee isolation at the language level you could have isolated kernel components entirely without hardware support.

Rust is close to allowing that with little performance overhead, close but no cigar.

[Nominal Animal]

With a programming which guarantee isolation at the language level you could have isolated kernel components entirely without hardware support.

It is not proper isolation, proper security: it is purely a theoretical option that does not provide any isolation guarantees in practice.

[Marco]

There are only guarantees in theory, hardware or software doesn't matter in that regard.

[Nominal Animal]

There are only guarantees in theory, hardware or software doesn't matter in that regard.

Uh, what?

A proper design guarantee is valid when the implementation has no bugs.
Yes, bugs do always exist; so, we layer security, so that typical bugs do not immediately grant full access to the entire machine.
Thus, a single guarantee is not what we should be looking at, but the entire structure or stack of guarantees.

A simplified example: the userspace-kernel layer verifies all userspace-provided parameters and scrubs all kernel-private information not intended to be passed back to userspace.  Let's say this promise fails, and a specific syscall fails to verify a pointer.  Assuming virtual memory is used, we rely on process isolation, so that no pointer value in one process can be used to refer to data in another process.  Thus, the vulnerability surface is limited to that buggy syscall, and exactly what the kernel does with the pointer.  Neither promise or guarantee alone is sufficient; for proper isolation, we need both.  In case of a bug in one, the security implications are limited.  So, root exploits require more than one bug to accomplish their task, which makes such exploits harder to implement, and yields better security expectations for the users.  Yes, it is all shades of gray, but it is not just one layer of paint, not at all.

You essentially propose to replace the entire graph or stack of guarantees with the single one provided by a programming language implementation. 

[SiliconWizard]

You essentially propose to replace the entire graph or stack of guarantees with the single one provided by a programming language implementation. 

But the good thing is that you're then replacing computer science and thought-out architectures with a silver bullet.

[Marco]

Yes, bugs do always exist; so, we layer security

Meltdown burned through all of it. The difference between what's a layer and what's a link is hard to predict.

Any way, kernels are monolithic despite decades of exploits ... not much layering going on inside them. Software only approaches have the advantage of not needing to upset the TLB with all the involved delays and they can run on processors without any MMU at all.

[DiTBho]

With a programming which guarantee isolation at the language level you could have isolated kernel components entirely without hardware support.

It is not proper isolation, proper security: it is purely a theoretical option that does not provide any isolation guarantees in practice.

I do strongly believe that "isolation" can only achieved by "hardware segregation".

[DiTBho]

You essentially propose to replace the entire graph or stack of guarantees with the single one provided by a programming language implementation. 

indeed the ICE_AI driven of my-c doesn't trust any stack of guarantees in any of the monads used in my-c, and, instead, checks everything at run-time; and even this way, there are some undetected and uncatched bugs and vulnerabilities still lurking around, hidden somewhere, waiting to bite you, and they will when you less expect it.

[Nominal Animal]

Yes, bugs do always exist; so, we layer security

Meltdown burned through all of it.

I disagree.  I consider only remotely exploitable privilege escalations as such.

Meltdown meant virtual memory protections are circumventable on certain processor architectures, unless relatively costly software workarounds are applied.  As such, it was a very serious local privilege escalation risk.  As a silicon design bug on more than one design, it was unusual and widespread, affecting all operating systems running on such hardware.  But it did not "burn through all security".

The difference between what's a layer and what's a link is hard to predict.

I don't follow.  What do you mean by 'link'?  By 'layer', I refer to each security promise or guarantee, as they are structured and rely on each other to provide the overall security in a system.

The first 'layer' is input filtering and buffer overrun protection in userspace applications.
The second 'layer' is process isolation and kernel-userspace barrier via virtual memory.
The third 'layer' is the kernel verifying parameters and scrubbing kernel-internal information in unused registers and userspace-visible structures when returning to userspace.

Each of these can be split into security sub-promises, of course.  While these are traditionally described as a stack, a graph would be more correct.

In Unix/POSIX systems, for example, it has always been typical that application binaries are not writable by normal users (without privilege escalation).  This means that even if you manage to inject code into a running user process, it is nontrivial for that process to infect other binaries.  This, too, is a security 'layer', only a purely userspace design one.
 _ _ _

There is no difference to the userspace security under monolithic and microkernels: all currently known designs can be implemented under both equally well.  The main security difference is that microkernels have internal privilege separation, similar to userspace processes, within the kernel itself; whereas monolithic kernels are more similar to a single userspace process.

Replacing all of that with the security promises a programming language provides is a ridiculous trade.

[Nominal Animal]

With a programming which guarantee isolation at the language level you could have isolated kernel components entirely without hardware support.

It is not proper isolation, proper security: it is purely a theoretical option that does not provide any isolation guarantees in practice.

I do strongly believe that "isolation" can only achieved by "hardware segregation".

I agree, because I have never heard of any way of providing similar strength/reliability of security promises, in software.

Yet, hardware segregation alone does not provide full system security; no single feature does.  It is a good start, though.
To do useful work, our kernels and userspace software needs the ability to bypass hardware segregation in specific cases: the kernel-userspace barrier and interprocess communication being the most commonly used ones.

You could say that security-wise, microkernels extend the security 'layering' (in the sense I defined in my previous answer above) into the kernel itself, making sure each part or subsystem of the kernel has only access to what it actually needs.  In monolithic kernels, all kernel code has equal access to all hardware and all userspace processes; there are no kernel-internal "security barriers".

Interestingly, many of the microcontrollers I use have rudimentary hardware segregation/protection, at least in the form of registers that can only be modified when a separate "lock" register contains a specific value.  (The more advanced ones also have memory access protections, even when they have no virtual memory support.)  Many support splitting the built-in Flash memory to a write-protected bootloader area, even when the rest of the Flash is updated.
In this sense, it is difficult to separate 'security' from 'robustness', because both are essentially about trying to fulfill the promise of "correct operation".

Since current general-purpose computer hardware can provide the hardware segregation with neglible overhead, it is obvious that a microkernel design can be more secure/robust than a monolithic one, simply because most of the necessary aforementioned holes in e.g. the kernel-userspace barriers don't open up the entire hardware, but only a small part.

[Marco]

I don't follow.  What do you mean by 'link'?  By 'layer'

An allusion to chain being as strong as its weakest link.

Meltdown burned straight through all read protection layers. The hardware protection was a link, not a layer.

[DiTBho]

@Nominal Animal
(quoting your points in a list)

• layer1: input filtering and buffer overrun protection in userspace applications
• layer2: process isolation and kernel-userspace barrier via virtual memory
• layer3: kernel verifying parameters and scrubbing kernel-internal information in unused registers and userspace-visible structures when returning to userspace

Specifically, my automatic tools check layer1 and layer3. Layer2 is not currently checked.

Plus, I have a seriously problems checking physical addresses to virtual addresses without triggering unwanted hardware reaction. So, at the moment, I am using a kind of "kgdb-like" (linux kernel gdb) software tool instead of the ICE-tap.

With MIPS, both cache and TLB are not friendly when you have to use invasive hardware tools to inspect them.

[SiliconWizard]

Just stumbled upon this video:

(This is not a promotion of this guy's channel, which I didn't know before.)

He goes through the famous discussion between Linus and Andrew. One can see that heated online technical discussions were already a thing 30 years ago.
I find it interesting to go through all the arguments in retrospect, the assertions that were made, and where we are now.

[Marco]

I agree, because I have never heard of any way of providing similar strength/reliability of security promises, in software.

Javascript containment in the major browsers is more on the level of VM containment than local process containment, they also use process isolation but that's ultimately the lesser of their security guarantees.

There's always software border guards, which are usually the weak point outside of corner cases like Meltdown. Escapes through privilege escalation using faulty APIs, VM escapes use faulty virtualization services running on the host, etc. With the right architecture, a pure software solution doesn't need many more LoC attack surface than process/VM isolation.

[ejeffrey]

Three major points that I see in favor of the "microkernel" approach - which I would actually like to see in software in general, not just kernels - are: 1/ isolation, 2/ modular designs with small, manageable chunks and 3/ message passing and much saner handling of resource sharing.

Now of course, what may have been problematic 30 years ago on 386 computers has become irrelevant to a significant extent.

Maybe.  But while computers have gotten dramatically faster, the relative cost of context switching has also gone up considerably because modern CPUs have so much more state and the mismatch between the ALU speed and the memory hierarchy have gotten worse.  Of course there are variations in microkernel design and how much context actually gets switched but I don't think the argument that the costs that were prohibitive in the 90s are now trivial is actually obvious.

Two other points to consider: computers are now "fast enough" for many but not all purposes and a bit of overhead is not always going to be prohibitive even if it is a significant fraction.

On the other hand, the force multiplier is much larger especially for one of the very few mainstream OS kernels.  If you can do a bit of development work to reduce context switches in IO and reduce overhead by 0.1% but you get to deploy that to a billion devices that's a big difference in aggregate. It still may or may not be worth it especially if you compromise correctness but it's worth considering.

[Marco]

The cost of TLB flushes went up. On the other hand PCID is now supported by both AMD and Intel.

[SiliconWizard]

Again, most current "mainstream" OS kernels are hybrid kernels (macOS, Windows.)
Linux is a bit an oddball with its monolithic structure. In that regard, Andrew was right. He wasn't so right in his claim that GNU Hurd would be the future. Yeah, not really.

But Linus's point was precisely to design something practical that would just work on commodity machines, and history quickly showed he was right.
It's a bit of a pragmatism vs. idealism approach.
Fun point about Andrew's "I wouldn't give you a good grade for this". So much for an OS that is running most servers, supercomputers and mobile devices in the world, that is decently secure and that has created probably directly and indirectly a multi-trillion dollar business worldwide over 30 years.

Pure microkernels do exist on well established systems, such as QNX.

More isolation would certainly be a good thing in any case. Yes, current hardware may still not be optimized for this, and possibly something to focus on first.

Heck, one thing that I still find mind-boggling, on a higher level, is using stacks that hold both data and return addresses. Merely changing that (which is not going to be a quick job though) would avoid a large chunk of exploits.