Author Topic: [solved] Linux: maximum bootable kernel size in PPC/32bit  (Read 6591 times)

0 Members and 1 Guest are viewing this topic.

Offline brucehoult

  • Super Contributor
  • ***
  • Posts: 2898
  • Country: nz
  • Formerly SiFive, Samsung R&D
Re: Linux: maximum bootable kernel size in PPC/32bit
« Reply #25 on: September 21, 2021, 11:38:22 am »
u-boot dropped support for PPC 4xx in 2017.
Linux dropped support for PPC 4xx with the 5.8.0 kernel.

Oh! This seems to be the first mention you're using a PPC 4xx!  Which one? Do you have an MMU? An FPU?

It seems Debian and Ubuntu had their last releases for 32 bit PPC Macs in 2015 (Debian 8, Ubuntu 14.04)


Hmm .. https://www.kernel.org/doc/html/latest/powerpc/cpu_families.html says 4xx is "BookE" and has a software reloaded TLB (fine .. so do some RISC-V) and are supported ... still?

I really haven't followed the PPC situation for more than 15 years since Apple switched. Used to love the things, especially from 604e and on. Wish I hadn't sold my dual 2.0 G5.
 

Offline DiTBho

  • Super Contributor
  • ***
  • Posts: 1758
  • Country: gb
Re: Linux: maximum bootable kernel size in PPC/32bit
« Reply #26 on: September 21, 2021, 12:34:08 pm »
Oh! This seems to be the first mention you're using a PPC 4xx!  Which one?

two boards, 405 and 440. At the moment I am working with the 405 one.

Do you have an MMU? An FPU?

TLB: yes
FPU: no, full kernel FPU emulation

The FPGA is memory mapped and offers fixedpoint in hw, but it's a raw hack at the moment

It seems Debian and Ubuntu had their last releases for 32 bit PPC Macs in 2015 (Debian 8, Ubuntu 14.04)

I have my custom builder, I can build 2021 stage4, even for PowerMac-G4 (PPC7450).
Gcc has special flags for 40x, but basically the PPC4xx-stage is profiled like a PPC750-softfloat stage.

4xx is "BookE"

yes, I forced u-boot to start the kernel with
- cache completely invalidated and flushed
- TLB disabled, so the kernel-image can be copied to 0x0000.0000
- PCI pre-programmed with all the devices forced into PCI native mode
- serial0 programmed in polling mode so it can output useful information

Wish I hadn't sold my dual 2.0 G5.

G5 is profiled as PPC64-BE. Its code is different; nowadays PPC64 is well supported for IBM POWER9 and POWER10.

PPC32 has several "platforms" of reference. in my case it's not "Apple" but rather AMCC and Xilinx, but unfortunately they dropped the support years ago.

The problem I have seems related to the Device Tree. There was no DTB in kernel 2.4 when we purchased the boards, DTS was introduced years later (>= k2.6.26), and it has always been a plague.

The 2.6.26 kernel was also the first "hybrid" kernel with both the "arch/ppc" and "arch/powerpc" branches merging into the latest "arch/powerpc". And that has always meant problems, loss of files and features (e.g. all the GPIO support for 4xx stopped to work, and got removed), loss of support and regression.

kernel-4.11.0: <7MB, boots
kernel-4.14.246: <7MB, boots, of_irq_parse_pci fails
kernel-4.14.99: <7MB, boots, of_irq_parse_pci fails
kernel-4.4.199: <7MB, boots, of_irq_parse_pci fails
kernel-5.2.1: >8MB doesn't boot
kernel-5.2.1: <8MB with linkaddr=0x0090.000 doesn't boot
kernel-5.2.1: <8MB with linkaddr=0x0080.000 boots, of_irq_parse_pci fails
kernel-5.2.1: <8MB boots, of_irq_parse_pci fails
kernel-5.5.1:  <8MB boots, of_irq_parse_pci fails
kernel-5.7.19:  <8MB boots, of_irq_parse_pci fails
kernel-5.8.0:  <8MB doesn't boot (4xx support dropped)

(edit: all tests performed with tftpload 0x0400.0000 (64MB))

I can boot modern v5 kernels (<= k5.8.0), but there is a serious problem with the maximum bootable kernel size due to a "bug" somewhere with the boot-wrapper, and it somehow looks to be related to the DTB stuff.
« Last Edit: September 21, 2021, 12:47:37 pm by DiTBho »
 

Offline DiTBho

  • Super Contributor
  • ***
  • Posts: 1758
  • Country: gb
Re: Linux: maximum bootable kernel size in PPC/32bit
« Reply #27 on: September 21, 2021, 09:26:50 pm »
Today I improved the early-console_out functions and I instrumented more code but still the kernel dies before the first message from the kernel shows up.

That's a very nasty situation to debug, and it's where things like "git bisect" are really useful just to narrow things down, which isn't an option here because *every* tested kernel after 2.6.26 (so *every* DTB-kernel) dies the same way when linkaddr=0x0090.0000.

There is nothing I can "git bisect". Too bad.

Kernel 2.6.15 can boot with linkaddr=0x0090.0000, but kernel 2.6.15 is not a DTB-kernel, and it's also "arch/ppc", so its code is completely different regarding PCI, how to initialize things and, etc.

It sounds something went bad back in 2005-2007 when "arch/ppc" was merged into "arch/powerpc", and no one ever noticed before (or no one ever cared)  :-//
 

Offline nctnico

  • Super Contributor
  • ***
  • Posts: 22988
  • Country: nl
    • NCT Developments
Re: Linux: maximum bootable kernel size in PPC/32bit
« Reply #28 on: September 27, 2021, 12:50:40 am »
Late to the party... Can you hack u-boot to add a (simple) memory verification command (if that isn't already there). I'd verify whether the memory you are trying to use actually works. From your test results I start to doubt that. Alternatively it is possible that the kernel overwrites itself and may need more space allocated for itself.
There are small lies, big lies and then there is what is on the screen of your oscilloscope.
 
The following users thanked this post: DiTBho

Offline DiTBho

  • Super Contributor
  • ***
  • Posts: 1758
  • Country: gb
Re: Linux: maximum bootable kernel size in PPC/32bit
« Reply #29 on: September 27, 2021, 09:29:11 am »
Can you hack u-boot to add a (simple) memory verification command (if that isn't already there).

Already done :D

u-boot reserves the first and the last 1MB for its internal structures, but all the tested ram from (0+1)MB to (256-1)MB is fine.
 

Offline DiTBho

  • Super Contributor
  • ***
  • Posts: 1758
  • Country: gb
Re: Linux: maximum bootable kernel size in PPC/32bit
« Reply #30 on: September 27, 2021, 09:42:11 am »
Alternatively it is possible that the kernel overwrites itself and may need more space allocated for itself.

Yes, it is possible, just as it is possible that during the boot sequence, just before jumping into kexec(), something goes wrong and the code that has to manage the flat device tree only sees 16 Mbytes at boot, which is enough with linkaddr = 0x0080.0000, but not enough with 0x0090.0000, and in this case that code for some weird reason can't map more ram(1).

That weird reason seems specific to ppc40x, a PPC750 doesn't exhibit this problem, but ... it's my speculation, and the code (assembly) is different.

It's funny, because if the code passes the point of no return kexec (), then it correctly sees 256MB of ram.

I mean, with linkaddr = 0x0080.0000 and a 7.5MB sized kernel, it works fine and "/proc/mem" correctly reports (256-8) MB available to the user


(1) it should arise some machine_exception at this point, but ... I cannot easily track it down because I don't have a jtag debugger. I need to hack the exception tables in order to put there something like a call to my "early_puts" so I can see something on the serial monitor, otherwise the machine simply silently dies without any clue.
 

Offline DiTBho

  • Super Contributor
  • ***
  • Posts: 1758
  • Country: gb
Re: [solved] Linux: maximum bootable kernel size in PPC/32bit
« Reply #31 on: September 27, 2021, 04:00:40 pm »
Solved  ;D ;D ;D

I asked to a PPC40x expert and we found an error in the TLB pre-initialization.
Rather PPC40x-specific.

 

Offline Monkeh

  • Super Contributor
  • ***
  • Posts: 7482
  • Country: gb
Re: [solved] Linux: maximum bootable kernel size in PPC/32bit
« Reply #32 on: September 27, 2021, 04:14:21 pm »
It wasn't something to do with systemd? The mind boggles.
 

Offline DiTBho

  • Super Contributor
  • ***
  • Posts: 1758
  • Country: gb
Re: [solved] Linux: maximum bootable kernel size in PPC/32bit
« Reply #33 on: September 27, 2021, 06:35:51 pm »
It wasn't something to do with systemd? The mind boggles.

LOL  :D
 

Offline brucehoult

  • Super Contributor
  • ***
  • Posts: 2898
  • Country: nz
  • Formerly SiFive, Samsung R&D
Re: [solved] Linux: maximum bootable kernel size in PPC/32bit
« Reply #34 on: September 28, 2021, 12:13:27 am »
Solved  ;D ;D ;D

I asked to a PPC40x expert and we found an error in the TLB pre-initialization.
Rather PPC40x-specific.

Can you explain it exactly?

In RISC-V land we used to get a fairly steady stream of people whose toy OS works fine on QEMU but fails on real hardware when they try to switch from Machine mode to Supervisor (or User) mode because QEMU was more lenient about PMP [1] initialization (if you didn't initialize PMP then it assumed you don't have one). A few months ago QEMU changed so that just like on most real hardware there is a PMP and you must initialize it. So now people's things that used to work on QEMU suddenly don't. But it's a whole lot easier to debug it there than have real hardware that silently dies.


[1] Physical Memory Protection. A hardware unit that provides RWX protection for typically 16 memory ranges without address translation. Even small microcontrollers usually have it if they provide User mode. At reset Machine mode can access everything and Supervisor and User mode can access nothing. On Linux (etc) machines it lets Machine mode lock even the Linux kernel(s) out of certain memory ranges. It works on physical addresses, not virtual addresses like the TLB (if you have one).
 

Offline DiTBho

  • Super Contributor
  • ***
  • Posts: 1758
  • Country: gb
Re: [solved] Linux: maximum bootable kernel size in PPC/32bit
« Reply #35 on: September 28, 2021, 01:32:07 am »
Can you explain it exactly?

u-boot operates without virtual memory. The same does the first part of the kernel loader (where I attached my early_puts for debugging). From here it goes quite specific, basically the assembly code for the TLB initialization was not correct and when the pre-booter tried to move into virtual memory it completely lost all references to everything outside its window.
 

Offline brucehoult

  • Super Contributor
  • ***
  • Posts: 2898
  • Country: nz
  • Formerly SiFive, Samsung R&D
Re: [solved] Linux: maximum bootable kernel size in PPC/32bit
« Reply #36 on: September 28, 2021, 01:40:42 am »
Can you explain it exactly?

u-boot operates without virtual memory. The same does the first part of the kernel loader (where I attached my early_puts for debugging). From here it goes quite specific, basically the assembly code for the TLB initialization was not correct and when the pre-booter tried to move into virtual memory it completely lost all references to everything outside its window.

Yes I understand that.

I just explained how people often have similar problems switching from using physical memory to virtual memory on RISC-V.

I'm asking what, exactly, was wrong with the TLB initialisation that works on PPC750 but not 40x. The actual instructions.
 

Online thinkfat

  • Supporter
  • ****
  • Posts: 2022
  • Country: de
  • This is just a hobby I spend too much time on.
    • Matthias' Hackerst├╝bchen
Re: [solved] Linux: maximum bootable kernel size in PPC/32bit
« Reply #37 on: September 28, 2021, 07:24:39 am »
Solved  ;D ;D ;D

I asked to a PPC40x expert and we found an error in the TLB pre-initialization.
Rather PPC40x-specific.

That would've been hard to debug even with a JTAG probe.

BTW, you wouldn't necessarily have to buy an expensive probe. OpenOCD has at least some support for PPC and should allow using a simple FTDI based probe.
Everybody likes gadgets. Until they try to make them.
 

Offline DiTBho

  • Super Contributor
  • ***
  • Posts: 1758
  • Country: gb
Re: [solved] Linux: maximum bootable kernel size in PPC/32bit
« Reply #38 on: September 28, 2021, 11:37:31 am »
I am not a PowerPC expert. I worked years ago for a company that had a few PPC based products (440, 460, 8xx and e500) and I happen to have a couple of PPC40x based hobby projects, and I also like to run Linux on my mac-mini/PPC

I'm asking what, exactly, was wrong with the TLB initialisation that works on PPC750 but not 40x. The actual instructions.

Not a single instruction, but a sequence of instructions.

On Linux, the critical code is segregated within a specific CPU module
  • PPC32
  • PPC64
  • PPC40x
  • PPC44x
  • PPC8xx
  • ...
each item in the list has a dedicated assembly module with low-level exception handlers, MMU support, trap, FPU-trap (if there is no physical FPU) and interrupt dispatch.

I haven't gone into the details, but PPC32 has different procedures with more support and interest. It needs to loads the BAT registers with the values set by MMU init, which for example takes care of whether we are on a 601 or not. These facilities have been exhaustively implemented with a rich set of checks, while the 40x code is poor, probably because it has less interest.

PPC32 is mainly "PMAC" (PowerMac), which assumes OF+bootX, while PPC405 assumes u-boot. There are differences even here, e.g. with the CPU already has the MMU on, and there is specific code to turn it off before changing BATs (MMU's registers).

PPC32 has the 1st slot of ram mapped with the bats and can map more slot on demand. Indeed when I tried to compile a 9MB kernel for my mac-mini/ppc, it correctly mapped 32MB to facilitate the kernel pre-booting. There is a procedure (in assembly) to check and manage things transparently. The PPC40x code does something similar but the procedure is different and it is simpler and less "error prone". It goes back to running unmapped so it can load up new values and change to using exception vectors. This way, all the CPU has to do is invalidate the TLB to clear the old set-TLB mappings, and to map an additional slot it requires to configure, reserve and load two entries into TLB slots. But it's not "on demand", but rather hard-coded in order to map the first 32 MBytes of memory 1:1, virtual to physical and more importantly it's how to set the cache mode.

The bug was that on the hard-coded part, it reserved only one slot instead of two, so it only mapped 16MB instead of 32MB and there was no checking for that.
 
The following users thanked this post: Nominal Animal


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf