Monolithic or microkernels?

[SiliconWizard]

Yet another debate. Not exactly meant to revive the oldish debate between Linus Torvalds and Andrew Tanenbaum (author of MINIX), although... why not.

Do you think Linus was right and microkernels make no sense from a performance POV, and the other alleged benefits are not enough to counterbalance it?

[DiTBho]

Do you think Linus was right and microkernels make no sense from a performance POV, and the other alleged benefits are not enough to counterbalance it?

I think Linus was right when he said it because microkernels consume more resources and when Linux took its first baby step (v0.0.1) ... we had i386SX25 with 8 Mbyte of ram

microkernels have always been superior on paper... but they weren't that great in practice... and if you look at thier legacy, well ... "Haiku" is based on NewOS Kernel, but it's a hybrid solution 

[eutectique]

I think Linus was right when he said it because microkernels consume more resources and when Linux took its first baby step (v0.0.1) ... we had i386SX25 with 8 Mbyte of ram

At the same time:

• OS-9 2.x was running on M68000 with 1 MB of RAM and 2 MB of flash
• QNX 2.x was running on i286 with 2 MB of RAM
• QNX 4.x was running on i386EX with 2 MB of RAM and 4 MB of flash

[mapleLC]

I think Linus was right when he said it because microkernels consume more resources and when Linux took its first baby step (v0.0.1) ... we had i386SX25 with 8 Mbyte of ram

At the same time:

• OS-9 2.x was running on M68000 with 1 MB of RAM and 2 MB of flash
• QNX 2.x was running on i286 with 2 MB of RAM
• QNX 4.x was running on i386EX with 2 MB of RAM and 4 MB of flash

Isn't containerized virtualization basically a useful implementation of this?

When these things are said, they can be in jest, but otherwise a limitation in the person's thinking at the time.  Bill Gates numerous times, but most famously saying that 640K of RAM ought to be enough for anybody.

I think the argument is settled, the microkernel has the utility it lacked when Linus made those comments.

[chickenHeadKnob]

I think Linus was right when he said it because microkernels consume more resources and when Linux took its first baby step (v0.0.1) ... we had i386SX25 with 8 Mbyte of ram

At the same time:

• OS-9 2.x was running on M68000 with 1 MB of RAM and 2 MB of flash
• QNX 2.x was running on i286 with 2 MB of RAM
• QNX 4.x was running on i386EX with 2 MB of RAM and 4 MB of flash

My experience with QNX in the 80's, version 1.x something and higher settled my mind on the performance issue. It ran fine on something as wimpy as 5MHz 8088 pc. I laughed at LInus's derisive they are just "message flingers" . Exactly, that is what is so good about them. For a modest performance hit you get great process isolation and network operation.

[alm]

I think the argument is settled, the microkernel has the utility it lacked when Linus made those comments.

Which practical application do you have in mind? Or are we talking about it being settled on paper? All current applications I can think of settled on hybrid approaches: Mac OS X used the Mach micro kernel, but then only has only one Mach service, so there's very little difference with a monolith. Windows NT started off as a hybrid approach, but since moving the graphic subsystem in kernel mode with NT4 is barely different from a monolith.

I think Linus was right when he said it because microkernels consume more resources and when Linux took its first baby step (v0.0.1) ... we had i386SX25 with 8 Mbyte of ram

At the same time:

• OS-9 2.x was running on M68000 with 1 MB of RAM and 2 MB of flash
• QNX 2.x was running on i286 with 2 MB of RAM
• QNX 4.x was running on i386EX with 2 MB of RAM and 4 MB of flash

Ancient operating systems run on ancient hardware (QNX 2.0 was from 1987, and I imagine OS-9 2.x was even older since the first release was in 1979). Especially operating systems targeting embedded platforms. I don't see what that proves. There were also versions of (monolithic) *nix that ran on the 8088.

[SiliconWizard]

One obvious issue with this "debate" is that one should clearly define what kind of "microkernel" one is talking about when making any kind of comparison.
There's a very general architecture, but beyond that it's very much down to each individual implementation.
Even MINIX is not quite what it initially was, and MINIX3 seems not to be too shabby.

While some microkernel implementations have "failed" in the sense of not getting really anywhere, QNX is clearly a successful implementation of the concept.
XNU, the macOS kernel, which is more identified as an "hybrid" between microkernel and monolithic, is a clear example that it also works, even for desktop use. To which extent its hybrid nature has helped it be a workable option rather than if it were a 100% microkernel, I admit I have no clue.

seL4 is also a current example of something that seems to work just fine.
https://sel4.systems/

[Marco]

Do you think Linus was right

No, message passing with pass by reference can be made fast enough to connect kernel components. See L4.

Address space separation within the operating system at the time was not an option, but that's not what defines a microkernel. Nowadays with PCID it's an option.

PS. device mapper is a good example of just how much function call indirection is doable for essential OS services. LVM based storage is already a giant pile of indirection, fast path message passing wouldn't break the bank.
It's not like microkernels make every function call a message, sane OS components don't have API call frequencies high enough to matter.

[Nominal Animal]

Do you think Linus was right and microkernels make no sense from a performance POV, and the other alleged benefits are not enough to counterbalance it?

Then, yes.  Now?  I can argue both ways, so it highly depends on the details like purpose, maintenance and development process, license, and programming language.  And target hardware, of course.

For heterogenous multiprocessing (different cores or distributed processors), microkernels fit much better than monolithic kernels.  We can see this in various computing sub-fields from phones and appliances (with dedicated cores) to distributed computing in HPC to server virtualization.

Really, the hardware has progressed much further than the programming languages we humans use to control the hardware.  To me, the culprit is not actually the programming languages themselves, but the limitations in human understanding and control of complex systems.  Synchronous vs. asynchronous is a prime example of this.  Most programmers, regardless of language or toolkit or development environment, still have issues in designing robust and correctly operating asynchronous systems.  Add in parallelism, and the set of programmers capable of fully understanding the system and avoiding the typical pitfalls (deadlocks, unnecessary exclusion via locks, "global locks") shrinks to a surprisingly small set.  To those who haven't done it themselves, and to those who do not understand that true development is a highly creative process, this seems to be near impossible to accept: they seem to believe any computer problem (design or not) to be solvable by simply throwing more mediocre programmers at it, until something sticks to the wall.

The kind of design architecture I believe future general purpose computers will need, is the kind where you can online and offline individual heterogenous processors and update kernel software in run time, without "rebooting" the system.  Something like cooperating microkernels, with no dedicated "boot processor", perhaps.  This belief is based on both the technological capabilities (inherent physical limitations in the capability of a single-core processors), and the projected/assumed/extrapolated needs of the human users, especially energy conservation, and high-bandwidth data flows.

[SiliconWizard]

Three major points that I see in favor of the "microkernel" approach - which I would actually like to see in software in general, not just kernels - are: 1/ isolation, 2/ modular designs with small, manageable chunks and 3/ message passing and much saner handling of resource sharing.

Now of course, what may have been problematic 30 years ago on 386 computers has become irrelevant to a significant extent.

[DiTBho]

• isolation
• modular designs with small, manageable chunks
• message passing and much saner handling of resource sharing

well summarized 

Now of course, what may have been problematic 30 years ago on 386 computers has become irrelevant to a significant extent.

Thinking about my IDT-R2K board (MIPS), comparing with my Atlas MIPS5++ prototype
- with MIPS-2K (as well as with m68k) there is only one ring of isolation, { superuser, user }
- with MIPS5++ and ARM there are more rings, and += hypervisor
- the MIPS5++ introduces the precious tr-mem, which simplifies and accelerate the message passing mechanisms

previously microKernels pushed the kernel into the "superuser" ring, all other services { network, filesystem, ... } in the "user" ring, and user applications in the same "user" basket, with some #trap to let them communicate.

#traps, especially on m68k (from 68000 to 68060), have a lot of overhead. MIPS has less, but trmem can help here, a lot.

Dunno about i386 

[Nominal Animal]

Three major points that I see in favor of the "microkernel" approach - which I would actually like to see in software in general, not just kernels - are: 1/ isolation, 2/ modular designs with small, manageable chunks and 3/ message passing and much saner handling of resource sharing.

Now of course, what may have been problematic 30 years ago on 386 computers has become irrelevant to a significant extent.

Quite.

As an example of the technical issues with isolation, consider a set of three processes A, B, C that circulate data.  A is only allowed to see what C sends, B is only allowed to see what A sends, and C is only allowed to see what B sends, so the three form a unidirectional cyclic graph.

You can implement this with three shared memory segments (A-B, B-C, C-A), with two of them mapped in each process.  Processing data involves copying the data from one segment to another, then signaling the next process that the chunk is ready.  Simple and robust, but copies data around.

Or, you can implement this using message passing with the data as one or more RAM page payload, with the kernel modifying the page virtual memory attributes.  This is the zero-copy approach, but involves quite a few caching-related details.  It is by no means guaranteed that this is faster than the copy approach, especially because this involves crossing the userspace-kernel privilege boundary (although so will cross-process locking primitives).  A particular corner case is B receiving a chunk from A, making a small modification, and forwarding it to C.  To be secure, A must not see the changes B makes to the data, and this can be difficult to ensure if A and B run on different cores.  (I don't even want to speculate on the issues if such transferred chunks contain executable code, and the interaction with speculative execution.)

It is the caching and cross-core synchronization wrt. virtual memory that makes the most theoretically interesting options not so simple in practice.  If performance is the key factor considered, then it is not at all obvious which one (monolithic or modular microkernels) is superior; it really depends on the hardware.

Isolation itself is an interesting subject, because while I like the old-style Unix user-group model, I could really use "sub-users" for process isolation and access control.  For example, if I start a process A, it would be very useful if A could start an isolated agent B that cannot directly access my private data, only data accessible to my own group; requiring A's authorization (on my behalf) for accessing my private data.   Similarly, it is not necessary for all kernel subsystems to be fully isolated from each other.  Permission tokens might be a way to implement something like this (as well as current models) more sanely, while giving robust security guarantees.

[SiliconWizard]

If you haven't, you could have a look at the MINIX(3) project. I'm not saying to actually use it, and have no opinion on the implementation, but many of the concepts it holds are exactly that.

[Nominal Animal]

If you haven't, you could have a look at the MINIX(3) project. I'm not saying to actually use it, and have no opinion on the implementation, but many of the concepts it holds are exactly that.

Sure.  For those who want to follow up on that, look at MINIX message passing (fixed-length messages with 56 bytes of payload) and memory grants AKA safecopies.

MINIX 3 is also POSIX conformant, meaning that even though it is a microkernel, software written in C to be POSIX-portable (say, across Linux/Mac/BSDs), should compile and run on MINIX 3 just fine.

[SiliconWizard]

I didn't know a whole lot about the details of the "argument" between Linus and Andrew (MINIX) until recently.

Apparently, the relatively strong reaction from Linus was not just about technical facts - it was, of course, but it's easy to understand it when you see that Andrew had been pretty bold by claiming in a famous 1992 note that "Linux was obsolete". Yeah, he started it!

Andrew was more generally claiming, back then, that monolithic kernels were obsolete and that the future would be microkernels.
At the time, he even cited Windows NT as a microkernel, although at the time it was merely speculation. Windows NT wasn't out yet. But it is known to be based, at least to a large extent in terms of architecture, on the MACH kernel. In the end, Windows NT is definitely not a microkernel, but it can be seen more like a hybrid kernel - like XNU for macOS - and in that regard, while his claim was pretty bold, even to this day, Linux is one of the few widely-used, general-purpose kernels out there that is purely monolithic.

[RoGeorge]

Well, if we count which one runs more copies today, decades later, I'm inclined to believe MINIX has more running instances than Linux, as we speak.  This is because any current Intel processor has a MINIX kernel inside. 
https://www.cs.vu.nl/~ast/intel/

[SiliconWizard]

Yes, IME is apparently using MINIX. Its author didn't particularly appreciate that Intel never cared to inform him about it.

[DiTBho]

I don't have to talk too badly, my file-system is much worse, but ... the Minix file-system, even if better than ext-v1,  is too catastrophe prone, and still needs to be improved: never compile it with large file caching!

(running Minix-v1 on my i486 laptop
it boots from a PCMCIA flash card)

[Marco]

It is the caching and cross-core synchronization wrt. virtual memory that makes the most theoretically interesting options not so simple in practice.

In your example, the kernels both have the exact same choices, the kernel copies the data or it remaps shared memory with all the TLB shenanigans involved (hopefully the architecture is sane enough the data cache can be left alone, with only the TLB cache being involved). The overhead of message passing is defacto irrelevant and they have to do the same things on the hardware, so it really doesn't matter.

[Nominal Animal]

It is the caching and cross-core synchronization wrt. virtual memory that makes the most theoretically interesting options not so simple in practice.

In your example, the kernels both have the exact same choices, the kernel copies the data or it remaps shared memory with all the TLB shenanigans involved (hopefully the architecture is sane enough the data cache can be left alone, with only the TLB cache being involved). The overhead of message passing is defacto irrelevant and they have to do the same things on the hardware, so it really doesn't matter.

I did not intend the preceding paragraphs to be an example of caching and cross-core synchronization, but as an example of issues with isolation.
I probably should have put a dotted horizontal line or something to indicate focus change.

(When you consider a message-passing-based microkernel and a monolithic kernel, yes, they both can utilize the same methods.  However, whether you can rely on all hardware being able to pass a reference across privilege boundaries (between different processes) and maintain privilege separation correctly, i.e. isolate processes reliably when doing large data transfers that way, is in question.   This means that while in theory, one could claim that say "message passing has less overhead, because it allows passing only the reference to the data, without copying the actual data", it is not generally/universally true in practice.  Marco is also right in that that statement would be silly/malformed/wrong, because monolithic kernels can do the same.  Switching the point of view to isolation, you see my point: isolation is much more complicated in practice than it sounds in theory.  It is also why I believe security has to be designed in to the very core, and cannot be added on top later on.)
 _ _ _

Note that overall in this thread, my focal point is that the practical details like hardware architecture and even programming language used, affects my opinion on which I would prefer, monolithic kernel or microkernel, even if using pure performance as my only measurement.

Currently, the hardest topics to 'grok' seem to be the interaction with privilege separation and caching mechanisms.  We see this in the relatively recent design bugs wrt. speculative execution.  If you follow e.g. the Linux kernel mailing list, caching-related bugs and hardware fixes tend to occur at a constant rate that only depends on the complexity of the caching architecture; with typical problems stemming from the fact that humans did not understand all the implications of the designed behaviour, rather than silicon bugs.
 _ _ _

If we harken back to the argument between Torvalds and Tanenbaum, one way to look at it (if we are neither of them ourselves) is to see it as former arguing based on practice, and the latter arguing based on theory and models.  Neither was wrong, per se, in my opinion, exactly because theory ≠ practice; and the argument itself was only possible between humans that place very different importance or weights to practical implementation vs. theoretical basis.

[Marco]

With a programming which guarantee isolation at the language level you could have isolated kernel components entirely without hardware support.

Rust is close to allowing that with little performance overhead, close but no cigar.

[Nominal Animal]

With a programming which guarantee isolation at the language level you could have isolated kernel components entirely without hardware support.

It is not proper isolation, proper security: it is purely a theoretical option that does not provide any isolation guarantees in practice.

[Marco]

There are only guarantees in theory, hardware or software doesn't matter in that regard.

[Nominal Animal]

There are only guarantees in theory, hardware or software doesn't matter in that regard.

Uh, what?

A proper design guarantee is valid when the implementation has no bugs.
Yes, bugs do always exist; so, we layer security, so that typical bugs do not immediately grant full access to the entire machine.
Thus, a single guarantee is not what we should be looking at, but the entire structure or stack of guarantees.

A simplified example: the userspace-kernel layer verifies all userspace-provided parameters and scrubs all kernel-private information not intended to be passed back to userspace.  Let's say this promise fails, and a specific syscall fails to verify a pointer.  Assuming virtual memory is used, we rely on process isolation, so that no pointer value in one process can be used to refer to data in another process.  Thus, the vulnerability surface is limited to that buggy syscall, and exactly what the kernel does with the pointer.  Neither promise or guarantee alone is sufficient; for proper isolation, we need both.  In case of a bug in one, the security implications are limited.  So, root exploits require more than one bug to accomplish their task, which makes such exploits harder to implement, and yields better security expectations for the users.  Yes, it is all shades of gray, but it is not just one layer of paint, not at all.

You essentially propose to replace the entire graph or stack of guarantees with the single one provided by a programming language implementation. 

[SiliconWizard]

You essentially propose to replace the entire graph or stack of guarantees with the single one provided by a programming language implementation. 

But the good thing is that you're then replacing computer science and thought-out architectures with a silver bullet.