[solved] how disturbing is x86? Unreal-Mode!!!

[DiTBho]

so, you have an architecture that was made to be CP/M compatible, and this explains why 8086 has "Real-Mode", but ... wait, what is "Unreal-Mode"?   

it has been mentioned here and there in Grub (legacy), and it looks to me like a hardware bug in the CPU design, so, it won't work on some CPUs(1), but when it works it's an exploit that you can use by setting the CPU in Protected Mode, then setting the descriptor cache's limits for your segment registers to any value higher than 64Kbyte, and finally setting back the CPU in real-mode, and ...

... and boom, the magic power of quirks breaks the 64Kbyte limit of real mode segments while retaining 16-bit instructions and the segment * 16 + offset addressing mode by tweaking the descriptor caches.

• code still limited to 64Kbyte
• data allowed accessing 4Gbyte
• CPU in real mode!!! 16bit!!!

it's useful when you're trying to load something that will run in 32-bit mode which is larger than what you are allowed to load in "conventional" (I mean without the Unreal Mode trick) Real Mode memory and you don't want to bother writing a protected mode driver yet, but you also want to avoid switching between real and protected mode to copy chunks from the conventional memory buffer into extended memory.




(1) tested on
- PC, AMI BIOS, intel 486-DX2 ---> it worked
- PC, Phenix BIOS, intel PentiumII ---> it worked
- AppleMini, Apple BIOS, intel Core2 dual ---> it worked
- Acord RiscPC/x86_guest_card, A1 BIOS, AMD 586 ---> it worked
- Acord RiscPC/x86_guest_card, A1 BIOS, Cyrix Cx486 ---> it didn't work!!! <---- Is it a problem?!?
- Acord RiscPC/x86_guest_card, A1 BIOS, Cyrix C5x86 ---> it didn't work!!! <---- Is it a problem?!?

untested on Xeon, Opteron, etc 


edit:
solved

[SiliconWizard]

Ouch. And people whine about Intel considering getting past all that.
 

[SiliconWizard]

Well, depends on how you look at it.
Their success, at least for a few decades, has largely been due to the great level of backwards compatibility. One can always claim to have been able to do better in retrospect, but that's just speculation.
Brain-dead engineers? Uh. A bit of humility never hurts.

As I said before, pretty much every time they tried to depart from that a bit too much, they failed. And no, that wasn't just because what they came up with was inferior. It's mainly because that's not what Intel's customers were looking for.
Trapped.

[DiTBho]

See how other people, those who know in detail these things, title the problem "A20 - a pain from the past", link  here

And it's referenced as "nonsense"

The 8088 in the original PC had only 20 address lines, good for 1 MB. The maximum address FFFF:FFFF addresses 0x10ffef, and this would silently wrap to 0x0ffef. When the 286 (with 24 address lines) was introduced, it had a real mode that was intended to be 100% compatible with the 8088. However, it failed to do this address truncation (a bug), and people found that there existed programs that actually depended on this truncation. Trying to achieve perfect compatibility, IBM invented a switch to enable/disable the 0x100000 address bit. Since the 8042 keyboard controller happened to have a spare pin, that was used to control the AND gate that disables this address bit. The signal is called A20, and if it is zero, bit 20 of all addresses is cleared.

Present
Why do we have to worry about this nonsense? Because by default the A20 address line is disabled at boot time, so the operating system has to find out how to enable it, and that may be nontrivial since the details depend on the chipset used.

When I updated my RiscPC to ROM v4.39 and faster companion x86 card, I spent 2 days fixing this nonsense for the x86-CPU guest card; the previous BIOS was incompatible with "!PC-v308", which was itself incompatible with the bew 586 guest cards (as they use different ASICs chip), and again, I found a dummy keyboard controller mapped to the bridge to allow DOS32 to see its 32Mbyte partition of the 128Mbyte system ram.


Anyway, the Unreal-Mode trick doesn't work with my Cyrix CPUs 

[james_s]

I feel like a broken record saying this, but backward compatibility is king, and nobody outside of a very rare few that do low level development cares what real mode is. Many really cool and innovative computers have appeared over the years, the Amiga (I have one) was fantastic, superior to the DOS PCs of the day but it couldn't run Lotus 123 or Microsoft Office. The Macintosh (I have several) was way ahead also, it could run Microsoft Office fairly early on but it was not affordable to the masses and it did not cater to game developers which everyone knows is what really drove the home PC market despite every kid saying they wanted one to do their homework. The BeBox was a neat and innovative machine but there was virtually no software for it so that one went nowhere. The NeXT was a neat machine, state of the art at the time but it had a workstation class price tag that made the Macintosh look inexpensive and again there was not really any software beyond what came with it. None of these offered backward compatibility with the established platforms and thus all of them failed in the market. As with smartphones, there is really only room for 2-3 platforms, anything else is doomed to fail and remain a niche. There is a chicken & egg problem where few are going to develop software for a platform that hardly anyone is using, and hardly anyone will use a platform that doesn't have a sizable library of software.

There's nothing "disturbing" at all about x86, it has a lot of baggage but that's because it has long offered one thing that no other platform can offer, and that is seamless compatibility with an enormous range of software. You could buy a 286 that could run almost everything you ran on your 8088, you could buy a 386 that ran almost everything that ran on your 286. You could buy a 486 that ran almost everything you ran on your 386 and so on. Given the historically high cost of software that was an *enormous* advantage and to a large degree it still is. For all but maybe 1,000 people on the entire planet, most of the x86 baggage is totally irrelevant, they never see any of it, most of the technical users never have to dive that deep and the non-technical users aren't even vaguely aware it exists and they don't care, and why should they? They just want to use their computer, they don't care how it works. It doesn't matter what whiz-bang innovations are lurking under the hood, if it can't run the software a person wants to use then it's useless. You must recognize that you are a part of an extremely small and exclusive club of people that ever touch any of the low level stuff and you cannot possibly expect the vast majority of people to care, it simply doesn't matter to them because they will never see any of the ugliness, they don't have a clue how a computer works internally nor are they interested in knowing and there's nothing wrong with that.

For some other platform to become the defacto standard, it will have to offer something tangible that normal people care about. For example it will have to be much cheaper than other solutions, or lower power, the latter is what has led to the success of ARM in the smartphone market, but smartphone users don't care about the gritty details of the architecture either, they just care about having a powerful device that fits in their pocket, goes a reasonable amount of time on a charge and runs all the popular apps.

[james_s]

Well, Professor Andrew S. Tanenbaum commented "if Intel had designed humans, we would have had some cr0 register to switch atavistic regression to chimp mode"

If anybody "designed" humans then IMO they are an incompetent moron. We have no upper boundary on the amount of fat we can store nor is there a point where we feel "full" and stop craving fat and sugar, it is literally possible to eat oneself to death. There are allergies that can be triggered and cause the body to harm itself, sometimes fatally trying to defend against harmless proteins. We have numerous vestigial features that have no known current use such as the appendix, sinuses, wisdom teeth, body hair and other bits and pieces. Other parts like the male prostate have a function but also almost always outlive their usefulness and become a liability. Our knees and hip joints are known weak points, the neck can be problematic, and there are many other issues.

Like x86, humans evolved over a long period of time from other organisms, and thus we carry a lot of baggage that no competent engineer would design into an organism today if they were starting from scratch. That doesn't mean humans are junk that should just be discarded to start over.

[Circlotron]

The original x86 architecture is like the width of two horses bums side by side.

[SiliconWizard]

These were all business decisions more so than engineering ones. Calling Intel's engineers brain-dead is completely silly.

Cost has certainly been a major concern back in the days and explains many of the mediocre solutions that had to be devised (but still allowed personal computing to exist at all in the end.)

I do agree with Tanenbaum's approach, but the irony is that x86 is still the main target of MINIX 3. Yes there's also an ARM port.

And certainly, the most popular solutions in the end are almost never the best ones from an engineering point of view.

What you called "just do it, make it cheap, and see how it goes" is pretty much what has been defining the computing business ever since it has become mainstream. Intel happens to just have been part of the beginning of this very business.

And sure since x86 is still almost everywhere, you have to deal with it.
Or stick to more niche products.

The good news is that the market is getting a lot more diverse. x86 is probably going to slowly die.

But actually most of what currently dominates in the computing industry is flawed, from software to architectures. In that huge mess, people keep devising band-aids just like Intel has done for several decades.

[Ed.Kloonk]

These were all business decisions more so than engineering ones. Calling Intel's engineers brain-dead is completely silly.

Cost has certainly been a major concern back in the days and explains many of the mediocre solutions that had to be devised (but still allowed personal computing to exist at all in the end.)

I hate to say it but honestly, kids these days scoffing at how 'dumb' we all were back in the day.

In the case of Intel and also those around at the time, these companies were pioneering pioneering technology, if that makes sense. A lot of it was guess work, some of it educated guess work but in reality, nobody really knew what the next big thing was going to be and when. A lot of companies lost a lot of money.

Nowadays, technology evolution is really down to who can time the market with the next leap forward and make it cost effective. To those who say how silly some of were back in those days, I put it to you how much be would have loved to have access to the information and tools available now. Some of us old farts are dismayed at how much possibility there is being squandered by the youth.

Having said that, I do enjoy the examination of these significant developments with all the hindsight.

[DiTBho]

These were all business decisions more so than engineering ones. Calling Intel's engineers brain-dead is completely silly.

really? Was the A20 decision, which is nonsense and causes a lot of trouble (read the article!) a business decision?
 

[DiTBho]

In the case of Intel and also those around at the time, these companies were pioneering technology

Motorola was a pioneering company as well as Intel, and if you look at m68000 there are issues as well. One regards the bus-error handling, which is unrecoverable by design (wrong), the other issue involves an instruction that leakages privileges between user mode and kernel mode (wrong).

Both are fixed with 68010, 68020, ... and note, 68k moved from 8bit (6800) to 32bit (68k) with the same problem you have in 80286: only 24 bits of physical addressing (68000), but note, the design was 32bit since the beginning, so this was not a problem when 68020 added 32 bits of physical addressing.

Intel ...
... let me understand, you have an 8086 CPU with 20bit of physical and logical addresses, and you think it's a good idea, while your CPU will grow over time and expand, to make a circuit in the next CPU, 80286, that disables some high address lines to make it back-compatible?

and, worse still, allowing a hardware bug like the Unreal Mode in every CPU they made? And since Unreal-Mode is known by software like DOS32 and Grub, why don't they consolidate it as a documented feature?


That's what I do find stupid and completely wrong.

[DiTBho]

the time I spent writing a bootloader.
- m68k: 25 hours (I hacked the ROM of my Apple LC475 in order to boot Linux on a 68060 CPU)
- PowerPC: 37 hours
- SH3: 51hours (I hacked my CASIO pocket graphing calculator in order to load custom math binaries)
- MIPS32: 40 hours + 32 hours to rewrite YaMon from scratch in assembly
- x86: ... 574 hours(1) and it's not yet 100% correctly working

"hours" means ... during free time, when the weather is terrible

anyway, 574 hours!!!!! that is what x86 { real-mode, unreal-mode, protected-mode, a20, ... } and all those crazy ideas mean in practice; plus, 30 hours only to setup two gdb sessions, one for the 16bit mode, and one for the 32bit mode, with glue code between them.

I just hope that I still don't have to work on the firmware of an x86-based industrial device, because if it's based on DOS16 that's one thing, but if they ask me to run different OSes, like DOS32 extension, or like VxWorks, or they pay Windriver for have support (and I am sure they won't), or will it be a similar experience to wanting to load Linux directly in protected mode.


(1) I did it wrong several times, even because the anatomy of Linux kernel binaries (bZInage, vmlinux.elf, plus the old and deprecated setup.elf) isn't entirely clear to me, the Grub and Lilo sources are tough to follow, and in between are a lot of cross-cutting issues like having a buggy PC-BIOS (Soekris ...) with services not working properly

OK, probably I would also better spend money, purchasing a  professional debugger instead of trying to develop it myself like I easily did for the other mentioned architectures.

[DiTBho]

The original x86 architecture is like the width of two horses bums side by side.

Yup, precisely 

Yesterday I found a shocking fact: although the technical manuals speak of 16-bit registers (e.g. AX is { AH, AL}), in real mode, you can still access all the 32bit registers (e.g. EAX contains AX) if you use a prefix to your "move" instruction.

So, all of the 32-bit registers are usable by simply adding "0x66" to the beginning of any instruction, and it seems to work even on Cyrix CPUs.

I don't know if it's a hardware bug, but I am happy it's there 

[dave j]

[magic]

Get UEFI hardware and forget about 16 bit modes

[DiTBho]

Get UEFI hardware and forget about 16 bit modes

eheheh I wish 
My Soekris net5501 is a 2006 design, and its Geode CPU is ~i586
So, it's pre-UEFI.

[SiliconWizard]

For those not familiar with the A20 issue: https://wiki.osdev.org/A20_Line

[David Hess]

Yesterday I found a shocking fact: although the technical manuals speak of 16-bit registers (e.g. AX is { AH, AL}), in real mode, you can still access all the 32bit registers (e.g. EAX contains AX) if you use a prefix to your "move" instruction.

So, all of the 32-bit registers are usable by simply adding "0x66" to the beginning of any instruction, and it seems to work even on Cyrix CPUs.

I don't know if it's a hardware bug, but I am happy it's there 

That is not a bug.  It is suppose to work that way.

[james_s]

In the case of Intel and also those around at the time, these companies were pioneering technology

Motorola was a pioneering company as well as Intel, and if you look at m68000 there are issues as well. One regards the bus-error handling, which is unrecoverable by design (wrong), the other issue involves an instruction that leakages privileges between user mode and kernel mode (wrong).

Both are fixed with 68010, 68020, ... and note, 68k moved from 8bit (6800) to 32bit (68k) with the same problem you have in 80286: only 24 bits of physical addressing (68000), but note, the design was 32bit since the beginning, so this was not a problem when 68020 added 32 bits of physical addressing.

Intel ...
... let me understand, you have an 8086 CPU with 20bit of physical and logical addresses, and you think it's a good idea, while your CPU will grow over time and expand, to make a circuit in the next CPU, 80286, that disables some high address lines to make it back-compatible?

and, worse still, allowing a hardware bug like the Unreal Mode in every CPU they made? And since Unreal-Mode is known by software like DOS32 and Grub, why don't they consolidate it as a documented feature?


That's what I do find stupid and completely wrong.

Well 68k machines were available, and they were superior in many ways except for two, and that is cost and compatibility, and thus they never achieved more than maybe 10% market share.

You seem to be either forgetting or deliberately ignoring the fact that all of these machines were/are produced for one reason and that is to earn money for the company selling them. So "you have an 8086 CPU with 20bit of physical and logical addresses, and you think it's a good idea, while your CPU will grow over time and expand, to make a circuit in the next CPU, 80286, that disables some high address lines to make it back-compatible?" Yes, I think it's a great idea, and it was very obviously the right business decision to make, because it resulted in an affordable machine that was much faster yet also capable of running the large library of existing software that people had invested substantial amounts of money in, and it resulted in selling vast numbers of these machines, earning the companies making them huge profits. Compatibility is king, it is the single most important feature followed by affordability, without it few will buy. If the 286 had not been code compatible with the 8086 and 8088 it would have been just another commercial failure, a footnote in the history of the PC revolution. Real mode was absolutely instrumental in its success and that of future x86 CPUs, backward compatibility is easily the single most important factor in their success and absolute dominance in the market for decades. When the 8088 was developed, nobody had any clue that computers would grow in power by leaps and bounds the way they did, there was just no way to see that coming. 8 bit microprocessors had been around for a while at that point and it really was not unreasonable at the time to believe the now famous line that "640k ought to be enough memory for anybody", I mean that was said at a time when 32k was a lot of memory, 640k was an absurd amount that no mere mortal could have afforded.

If you have a business is it smarter to go on an ideological crusade and develop something that is technically superior, or to produce an arguably inferior product that offers something that is of the utmost importance to potential customers, backward compatibility with their existing software, and thus much greater sales? All of the companies that tried the former failed, that should tell you something.

[magic]

Keeping the trolls well fed...

[DiTBho]

That is not a bug.  It is supposed to work that way.

Thanks! My doubt comes from the comments in a boot.S file (grub or Linux, I don't remember now), and it was a bit shocking the first time I opened it due to several comments telling the user how to force the assembly compiler.

something like if you put this prefix, you can access the 32bit EAX

Code: [Select]

[16 bit]
check_bz:
    pusha

    ; Test magic number of the BZ header
    mov     eax, dword [hdraddr + 0x202]
    cmp     eax, 'HdrS' ; <-------------------------- yeah, 32bit, it works!
    mov     dx, msg_sys_error1
    jne     panic

    ; Test jump instruction is there
    mov     al, byte [kernseg * 0x10]
    cmp     al, 0xeb
    mov     dx, msg_sys_error2
    jne     panic

    popa
    ret

Interesting, anyway, it's *VERY* useful for things like this!
I mean, it makes things simpler.

[DiTBho]

(
"nasm" can automatically put the "0x66" prefix in every "move" and "compare", etc, instruction if you mention a 32bit register as the argument

nice 
)

[David Hess]

Going by memory, each code segment has a bit which specifies the default data width, which can be 8/16 or 8/32.  When the data width prefix is used, then the other data width is selected for each instruction.

So in real mode the default is 8/16, and when the prefix is used, the 32 bit data width becomes available.

The danger is that while this makes the 32-bit registers available, a 16 bit operating system is unlikely to save the 32-bit register state so only that program can make use of them.  This problem also sometimes comes up with vector register extensions.

[james_s]

Dude .... 

I HAVE a business, I said several times that I develop ICEs and debuggers, plus other stuff, and if x86 takes me 500 hours to develop something, while other architectures/platforms take 50 hours, I think I am right to say that x86 is bad, especially if customers don't pay enough!

(and they don't)

We are in the "programming section" of the forum, it's supposed to talk about software/hardware problems (the title is "Unrel Mode"), but instead, you see and focus on "ideal" crusades for which you have to feel to defend (with x86 fanboyism) something with zero technical background just to see if your useless bullshit can make your life better.

Frankly, it's a troll attitude, and if last time I gave you the benefit of the doubt, now I have enough: stay on my ignore list!

p.s.
you also like to offend people personally, which LOL, it's fine with you that I don't give a shit about filing lawsuits against random people on the internet as I have many more interests in my life that spending my time making bad blood with people like you.

That's quite the word salad. You don't have a business selling CPUs back in the 80s and 90s do you? Because that's specifically what I was referring to. Your current business of writing debuggers is irrelevant to that discussion, nobody back in 1982 when the 80206 was released was thinking "oh yeah I bet there's gonna be this guy 40 years from now writing debuggers, let's really piss him off by designing the CPU in a way that will seem totally stupid at that time because that will be hilarious". That was not even a consideration at all. You're ranting and raving (how many threads have you started now?) and essentially calling the engineers that designed these chips decades ago bumbling idiots for doing things the way that they did, and I'm simply explaining that they were not idiots, in fact there were very good reasons they designed them the way that they did, it made good sense at the time, and I outlined what some of these reasons are.

I wouldn't call myself an x86 fanboy, I've never claimed that they are the best CPU architecture from a technological standpoint, indeed I openly acknowledge that there are other CPU architectures (several of which I own and use) which came along later that are technologically superior. I have merely explained why x86 chips have been dominant for so long, and defended many of the choices that made them the way they are and tried to explain why those choices made sense at the time and why in many cases they still make sense because compatibility with the existing software library is important to most users. I also have tried to explain that you can't reasonably expect the world to share your hatred for x86 because for the vast, VAST majority of users its warts are irrelevant. Not to you, I get that, but you are doing something that very, very, very few other people ever do. If you choose to take this personally that's on you, but you've started multiple threads ranting about x86 which frankly have the appearance of being a bit unhinged. I get that it makes your life more difficult, I get that you wish everyone was using something more modern, it would make your job easier, but right now as things stand it would make everyone else's life harder. You simply cannot expect people to spend more on hardware that does less for them because it can't run all the software they use.

Now I don't know why you even brought up lawsuits, if that's some kind of thinly veiled threat then it's a rather idiotic one because the courts in your country have no jurisdiction in mine. While I'm not familiar with UK law, my comments while not directly intended to offend you, are protected free speech by the laws of the USA, and legally speaking you haven't got a leg to stand on, you can't sue somebody for offending you even if it was deliberate. If I told some incriminating lie about you then under exceptional circumstances you may be able to sue me for slander, but I have not done any such thing so cool it and stop taking everything personally. Whether you have better things to do than sue random people that rubbed you the wrong way on an online forum, or you are the most litigious person on the planet suing anyone you possibly can is absolutely irrelevant to me, you couldn't sue me even if you wanted because I have broken no laws and I'm not a citizen of your country nor am I located there so none of your laws apply to me. You said you were adding me to your ignore list weeks ago so this shouldn't even be an issue.

[magic]

LOL

While I'm not familiar with UK law, my comments while not directly intended to offend you, are protected free speech by the laws of the USA, and legally speaking you haven't got a leg to stand on, you can't sue somebody for offending you even if it was deliberate.

You can't sue somebody for offending you. He can