[solved] how disturbing is x86? Unreal-Mode!!!

[DiTBho]

from commit 4bb3c7a0208fc13ca70598efd109901a7cd45ae7
Paul Mackerras, Mar 2018
KVM: PPC: Book3S HV: Work around transactional memory bugs in POWER9
To account for this different POWER9 chips have TM enabled in different ways.

POWER9
On POWER9N DD2.01 and below, TM is disabled. ie HWCAP2[PPC_FEATURE2_HTM] is not set.

On POWER9N DD2.1 TM is configured by firmware to always abort a transaction when tm suspend occurs. So tsuspend will cause a transaction to be aborted and rolled back. Kernel exceptions will also cause the transaction to be aborted and rolled back and the exception will not occur. If userspace constructs a sigcontext that enables TM suspend, the sigcontext will be rejected by the kernel. This mode is advertised to users with HWCAP2[PPC_FEATURE2_HTM_NO_SUSPEND] set. HWCAP2[PPC_FEATURE2_HTM] is not set in this mode.

On POWER9N DD2.2 and above, KVM and POWERVM emulate TM for guests (as described in commit 4bb3c7a0208f), hence TM is enabled for guests ie. HWCAP2[PPC_FEATURE2_HTM] is set for guest userspace. Guests that makes heavy use of TM suspend (tsuspend or kernel suspend) will result in traps into the hypervisor and hence will suffer a performance degradation. Host userspace has TM disabled ie. HWCAP2[PPC_FEATURE2_HTM] is not set. (although we make enable it at some point in the future if we bring the emulation into host userspace context switching).

POWER9C DD1.2 and above are only available with POWERVM and hence Linux only runs as a guest. On these systems TM is emulated like on POWER9N DD2.2.

Guest migration from POWER8 to POWER9
will work with POWER9N DD2.2 and POWER9C DD1.2. Since earlier POWER9 processors don't support TM emulation, migration from POWER8 to POWER9 is not supported there.

So, with IBM POWER we know TM is a problem, and we somehow know how to handle it, but I have never seen anyone from the IBM department, not even their traders, doing "sneaky innovation" propaganda when they have nothing but bugged features!

Intel does it, and it's frankly annoying!

[DiTBho]

I have migrated my FreeBSD router and NAS between multiple generations of x86 hardware

you could do this because there are layers and layers of compatibility software, just like I've already pointed out for Grub and Lilo that work at the cost of putting several orders of magnitude higher the number of lines of code that would be needed!

Which is annoying for devs and maintainers!

[Monkeh]

I have migrated my FreeBSD router and NAS between multiple generations of x86 hardware

you could do this because there are layers and layers of compatibility software, just like I've already pointed out for Grub and Lilo that work at the cost of putting several orders of magnitude higher the number of lines of code that would be needed!

Which is annoying for devs and maintainers!

Apparently less annoying than doing it for ARM SBCs. Or just far, far more worthwhile because there are many more useful x86 machines in the world.

[DiTBho]

You can't even migrate a disk with Linux installation from one SBC to another

if we talk about binaries, within the same processor family and endianness, the "rootfs" is rarely a problem unless compiled with implementation-specific stuff.

the real problems are
- firmware
- hypervisor
- kernel
- kernel modules (drivers)
- Device Tree (specifically, maps, and constraints)

look at the level of hardware standardization in ARM SOCs

Within each family people try to be as consistent as possible ... then, yeah, we have companies like Allwinner ... who do not even really contribute to GNU/Linux and/or U-boot and even at designing hardware make things all a mess up.

But whose fault is it?

Say, well, in Allwinner(1) it's not that they are so bright, and MediaTek(1) is a bit better but not that much, but for sure the Intel propaganda does not help here, even now telling people to buy Atom instead of ARM SBCs because it reduces the budget companies can allocate to better design and develop their products.

(1) I know these two well because I have worked with several of their SBC products.

[DiTBho]

Apparently less annoying than doing it for ARM SBCs

oh, yeah, it's really less annoying than doing it for ARM SBCs!!! In fact, it took me 1/4 of the hours to fix the Neo  (ARM) SBC, get my hands on both the kernel and u-boot, and work on less trivial stuff than the number of hours I spent on the AMD/Geode of the Soekris router to make a simple bootloader, having then to modify the Linux kernel to directly boot in protect-mode, as the default bZimage requires to be booted in real-mode, which. due to the 8086-platform constraints, is all a mess as you have to move things within the first 1Mbyte, plus a lot of nonsense things to handle BIOS incompatibility, all of this in assembly, and with a space constraint!

Really WOW!

Nah, let me say, it's okay for you to take, "turnkey" (ready out of the box), things that have already been done by the devs (this happens because x86 is mainstream due to its popularity), if you had to get your hands on it (which usually happens for industrial applications) ... you wouldn't be so happy to work with Intel/Amd.

First, because you have to read and study a thousand pages, even for simple things.
Second, because it is really difficult to give an estimate of the deadline both to the customers and to oneself for the calculation of the hours, and therefore we often end up "working practically for free".

And good luck debugging layers of layers of layers of compatibility things 

[SiliconWizard]

I'm afraid I (as most of us) would tend to focus on details while missing out on the big picture.

And precisely you focus on details while missing out on the big picture:

Uh? I didn't see this coming.
Humility is probably not one of your strong points.

I am just saying -1- do not buy x86 only because it's the most popular when you can buy something else, and -2- inform yourself before pointing the finger only at the most popular solutions!

That's a very general statement that can be said for almost anything. And it does look sad in a way. Now I'm guessing you're addressing this to the world in general - you can't hold that against me as I've been repeatedly talking about alternatives for various different topics rather than always going for what's trendy and what everybody else uses just because it's popular.

That said, what I've been trying here is to say that there is usually a reason why such decisions are taken, and you are refusing to understand those reasons. And the reasons are not just all because everyone is a stupid moron. It's a bit more complicated than this. If you had just said what you said right above, I don't think your thread would have triggered the reactions it did.

I've refered you (and anyone interested) to a book that talks about it in details and that I find pretty interesting. Basically again, innovation can't really happen in a large established corporation. Pretty much all people that had good ideas that turned out to be great innovations made them happen after quitting their job and starting their own company, or joining a small one. Just the way it works.

Understanding and even accepting a situation doesn't mean that you have to agree with it. But just keep in mind - as you have probably noticed over the years - that fighting a reality, especially in the business world, comes at a cost.

As to:

Second, because it is really difficult to give an estimate of the deadline both to the customers and to oneself for the calculation of the hours, and therefore we often end up "working practically for free".

Yep, that is already true for software development in general. Estimation is a very hard thing to do right - many argue that it's impossible.
I feel the pain, but what you do in such a case is that 1/ either you refuse the project altogether (again if you can afford to) or 2/ you manage to get paid hourly (which the client may not want, in which case, the client will refuse and you're good), or 3/ you make your best estimate and multiply it by 2 or 3. Probably 3 or 4 in your case for anything x86 related.

As a general rule, computing systems can be more or less defined as a triangle (a bit like the concept of the project management triangle): hardware, low-level software, and application software.
One of the vertices always has to give in. It tends to turn around a bit over time, but in most cases, it's the low-level software side of things that has to be the adjustment variable (and thus has to suffer).
So as another example, while writing application software on Windows has always been rather easy with a good level of backward compatibility, writing device drivers for Windows is close to a nightmare. (I think it has become a bit easier, I think mainly because you can now write many drivers as just filter drivers. But I still wouldn't want to have to touch that ever again.)

Regarding ARM, the issues are different. Calling ARM rubbish as magic did was extreme, but he probably wanted to make it sound a bit trollesque (as he likes to) while mimicking your own bold assertions. Now the issues have more to do with fragmentation, which is precisely what you tend to get over time when you don't/can't favor compatibilty. You can't have it all.

In that regard, RISC-V with its heavily modular ISA runs the risk of ending up even more fragmented. While I fully support RISC-V, mostly in the sense that it's an open-source ISA and allows anyone to design their own CPU, royalty-free and with no potential licensing issues ever (which opens a world of new possibilities), I don't know how the fragmentation problem is going to get solved if it's to be used for general-purpose computing (unless it gets gobbled up by a big fish which will dictate a standard all on its own, which would not be good news.)

[Nominal Animal]

Just a note for everyone: precise communication in linear human languages is difficult.  Moreso, when it is not in ones native language.

In my opinion, this thread is veering too close to the 'analyzing the tone' or 'reading between the lines' domain, which is not really useful.

When I catch this happening to myself, I combat it by re-reading the entire thread as if it was by other people, in order of posting.  It is illuminating, and also gives 'distance' so one is less likely to mistakenly take things personally.

[DiTBho]

In my opinion, this thread is veering too close to the 'analyzing the tone' or 'reading between the lines' domain, which is not really useful.

probably yes, but I prefer to give up.

[ejeffrey]

I have migrated my FreeBSD router and NAS between multiple generations of x86 hardware

you could do this because there are layers and layers of compatibility software, just like I've already pointed out for Grub and Lilo that work at the cost of putting several orders of magnitude higher the number of lines of code that would be needed!

Which is annoying for devs and maintainers!

Apparently less annoying than doing it for ARM SBCs. Or just far, far more worthwhile because there are many more useful x86 machines in the world.

It's relatively little to do with x86 vs arm, it's the system standard.  It's "IBM PC compatible" vs nothing.  ARM system makers make basically no attempt to support any system level standard.  A new PC today can boot up using bios emulation and legacy devices that emulate the keyboard, display, and serial ports from the 80s.  Then it can use a combination of probing for well known devices, looking up ACPI tables, and PCIe enumeration to find all the relevant hardware settings it needs to operate.  ARM has none of that other than possible PCIe, but on chip peripherals connected via AXI do not support plug and play enumeration.  Even very common peripherals are often located at random addresses.  Every ARM os has to be customized for the particular hardware platform it's running on, with a custom device tree to tell the bootloader and OS where everything is.

So yes the PC standard is crufty with lots of layers but the alternative isn't a better designed platform with nicer APIs for system configuration , it's nothing.

The ARM approach is nice for fast booting of fixed configuration platforms and deterministic device naming.  It's super inconvenient for having a modular system that you can download an OS for and boot it up.

[james_s]

That said, what I've been trying here is to say that there is usually a reason why such decisions are taken, and you are refusing to understand those reasons. And the reasons are not just all because everyone is a stupid moron. It's a bit more complicated than this. If you had just said what you said right above, I don't think your thread would have triggered the reactions it did.

This is the main thing that has bothered me from the start here, I feel a need to defend a lot of really smart people that developed all these things way back in the early days of the PC revolution, people that are being called morons that are not here to defend themselves or explain why they did things the way they did, and even if they were here, I strongly suspect the OP would simply ignore everything they said and continue to argue from a point of having 40 years of hindsight. It's one of those "debates" where the person doesn't actually want to debate, they have started with a conclusion, cherry picked only tidbits that support their conclusion, ignored everything else and then argue that everyone back then was stupid and they would have had the foresight to do better.

[james_s]

So yes the PC standard is crufty with lots of layers but the alternative isn't a better designed platform with nicer APIs for system configuration , it's nothing.

The ARM approach is nice for fast booting of fixed configuration platforms and deterministic device naming.  It's super inconvenient for having a modular system that you can download an OS for and boot it up.

Yes just look at the Raspberry Pi situation for example. The supply chain issues continues to make those virtually impossible to get, there are a handful of potential alternatives but none of them are compatible, I can't just take the OS build for my home automation system for example and pop it in some other arm SBC and expect it to boot up and work. On the other hand if Lenovo is sold out everywhere I can buy Dell, HP, or any number of other makes, install Windows or Linux and off I go. They're standardized and more or less interchangeable. There's huge value in that.

The cruft in the PC standard just doesn't matter in most cases, it's only a pain for a very small handful of people doing specialized low level work, for everyone else it is essentially zero cost, it's buried behind the walls, they never see it, there's no reason for them to care what it looks like.

[ejeffrey]

Also, a few people have, from time to time, made systems with x86 processors but most of the IBM PC "cruft" removed.  Among the most notable were the SGI Visual Workstations.  I don't think there is any world in which extremely expensive SGI branded workstations with x86 CPUs and running Windows were going to be a commercial success, but they also had the distinct problem that they could only run a specific modified versions of Windows NT.  Once booted it looked mostly like a regular PC at least from the application level, but you couldn't just boot a standard OS that worked on every other x86 system out there.

[james_s]

The SGI and Sun workstations were incredibly cool machines, I used to have a SGI Indy and an Indigo, and I still have several Sun workstations. The engineering that went into those is incredible and at the time they were very powerful, but even if they had not been vastly more expensive than IBM compatible PCs they never would have succeeded in the broader PC market because they were not compatible with the commodity software that most people used. They were specialized systems for specialized uses.

[David Hess]

It's relatively little to do with x86 vs arm, it's the system standard.  It's "IBM PC compatible" vs nothing.  ARM system makers make basically no attempt to support any system level standard.  A new PC today can boot up using bios emulation and legacy devices that emulate the keyboard, display, and serial ports from the 80s.  Then it can use a combination of probing for well known devices, looking up ACPI tables, and PCIe enumeration to find all the relevant hardware settings it needs to operate.  ARM has none of that other than possible PCIe, but on chip peripherals connected via AXI do not support plug and play enumeration.  Even very common peripherals are often located at random addresses.  Every ARM os has to be customized for the particular hardware platform it's running on, with a custom device tree to tell the bootloader and OS where everything is.

So yes the PC standard is crufty with lots of layers but the alternative isn't a better designed platform with nicer APIs for system configuration , it's nothing.

The ARM approach is nice for fast booting of fixed configuration platforms and deterministic device naming.  It's super inconvenient for having a modular system that you can download an OS for and boot it up.

Linus Torvalds has made that point before.  ARM lacks a standard for enumeration so each system requires a customized OS.  Well, there *is* a standard for ARM, but nobody implements it.

Note that nothing prevents fast booting with enumeration, and some x86 PCs support this, however there are fixed minimum timeouts on some legacy devices, for instance hard drives which must spin up, which cannot be shortened.

[james_s]

Note that nothing prevents fast booting with enumeration, and some x86 PCs support this, however there are fixed minimum timeouts on some legacy devices, for instance hard drives which must spin up, which cannot be shortened.

I see no reason these delays couldn't be bypassed if no legacy hard drives are found to be present though. Even a lot of older legacy systems that are still in service have been upgraded with modern SSDs.

[David Hess]

Note that nothing prevents fast booting with enumeration, and some x86 PCs support this, however there are fixed minimum timeouts on some legacy devices, for instance hard drives which must spin up, which cannot be shortened.

I see no reason these delays couldn't be bypassed if no legacy hard drives are found to be present though. Even a lot of older legacy systems that are still in service have been upgraded with modern SSDs.

Some commercial and consumer grade BIOSes, and many embedded x86 BIOSes, support exactly that.  For example Coreboot:

- Extremely fast
   - coreboot is designed to do the minimum amount necessary and get out of the way.
   - Especially for servers, coreboot provides faster boot up.
   - Desktop and Notebooks boot up in 400ms - 2.5s.
   - Server boot time can be as low as a quarter of the OEM BIOS boot time.

Thinking about it now, my relatively recent laptop does BIOS POST, BIOS enumeration, and starts and finishes booting Windows before my desktops even get through part of their BIOS POST and enumeration.  My laptop even gets to its Windows login prompt before my Android phone gets to its security login.  I think recovering from standby mode on my laptop might actually take longer, although then there is more startup stuff after logging in to complete.

[james_s]

There are cases where faster booting can be nice, but in practice I don't really care how long any of my devices take to boot, I boot them very rarely and normally just let them go to sleep. No matter how fast they boot it takes me much longer to get everything I was working with opened back up.

[Bruno_PT]

Apple competes with various system makers, but they do not compete directly with Intel or AMD.

Talking about data centers and scientific simulators, the fastest supercomputers today are built using Arm and POWER9, not x86 chips, why? well, have you seen the last Intel and AMD CPUs?


(Intel Xeon Platinum 8468)

Sure, they are able to outperform the chips made by Apple, Ampere, and NVidia, but only at the cost of insane power usage!
When you have a 20Kwatt cabinet it makes a BIG difference!

Lets check the top 500 then to see if that is actually true:
from www.top500.org/statistics/list/
x86 - 88% of the share
ARM - 1.6% (FUJITSU A64FX)
POWER9 - 1.4%

The Green 500 is equally dominated by x86, because even if intel isn't doing so good, Amd is doing a fenomenal work in the server space.

[DiTBho]

The Green 500 is equally dominated by x86, because even if intel isn't doing so good, Amd is doing a fenomenal work in the server space.

well, I just visited a couple of studios where they do professional video editing, and guess what they told me?
They replaced all thier x86/64 and AMD computers with "mac-mini studio", so M2 Pro chip, because

• Premiere on x86/64 is good, but Premiere on Apple Silicon is not optimized
• DaVinci is optimized (software-performance side) on both platforms, however on x86 machines it consume (electricity-performance side) 6 times the electricity of an M2 [/b]to produce the same results[/b]

Ampera is doing a fenomenal work, however, when you look at their budget, well ... Apple' shares are no longer worth a few pennies per share, they are worth hundreds of dollars, so much so that Apple is one of the companies with the most capital in the world, which can invest in research and development.

POWER is partly financed by DARPA and partly supported by the IBM business, which however makes more money with mainframes, and therefore not many financial resources are allocated. Something, but not enough compared to Apple, Intel and AMD.

And Ampera, in comparison, is a very small island with very little capital. Always compared to Apple, Intel, AMD, and IBM.

So, the only big competitor to intel/Amd x86/64 is Apple Silicon, and what do I think frankly?
Well, some companies still buy x86/64 only for a matter of contracts and ties to software that is x86/64-only.

Personally, if I had to pay myself the electricity bill out of my own pocket, and could run my pleasure GNU/Linux, rather than BSD/freebsd, and could recompile the programs myself (as I do with Gentoo, so ... no bin-blobs in none of packages), then I certainly wouldn't find spending money on smart way 6 times as much electricity to have x86 server in the computer room.

oh, then I have to say that, in my opinion, it's strange that Adobe didn't want to optimize Premiere for Apple Silicon, and this makes me think that they did it, in agreement with Intel/AMD, just as a "backroom deal" to try to sell more x86/64 chips.

I think this because DaVinci is open source software and it didn't take them long to optimize it. There are no technical problems when migrating sources from Windows10 to { MacOS, Linux }

Adobe didn't do it because they didn't want to do it! And for this reason, there are people who still buy some x86 machines, since Premiere on Apple Silicon sucks a lot, indeed people use FinalCut when it's possible!