has anyone ever worked with DeviceTree? (PowerPC, ARM)

[legacy]

so, I am working on Linux source (from 2.6.26 to 4.11), and noticed they are full of comments: problems with PCI, most of them located in arch/powerpc.

At some point, they decided they wanted to move to "DeviceTree". Therefore since 2.6.26 we have it, and code changed in a way that ... you have legacy code (things before 2.6.25), and current state of art (things after 2.6.26)

PowerMac still seems to be a stable branch since it depends on OF, so it doesn't require particular care on Device Tree code, everything is already built-in in their OpenFirmware PROM, which provides a decent "early initialization" of the PCI.

Completely different story with embedded boards, like ones have uboot in their prom. In this case uBoot misses special code, as well as linux requires a "boot wrapper", which must be provided and it's platform-specific.

It happens the PCI-code now contains common routines for ppc32 and ppc64 platforms, including PowerMac (PPC32) and pSeries and iSeries brands (PPC64), but for PPC32 machines the probing of PCI controllers from OF_Platform (open firmware) is currently recycled from the 64 bits only, mostly due to gratuitous differences between the 32 and 64 bits PCI code on PowerPC and the 32 bits one lacking some bits needed here.

It's a mess, at the end, it also requires specific code (through patches), which is hard to be maintain so it's often missing if there isn't enough demand for a board.

As consequence, a lot of embedded boards are doomed to stay code-broken until someone has the occasion ( is payed to ) fix things.

Also starting from kernel 3.* they have removed a lot or pre-initialization, assuming the kernel must start with all the hardware already pre-initialized. Well, uBoot v1.4 doesn't. It's not as smart as OpenBIOS or as OpenFirmware.

The chicken or the egg causality dilemma is commonly stated as "which came first: the chicken or the egg?"

Who has to provide the PCI pre-initialization? uboot? or the bootwrapper in linux kernel ?

None of them (in kernel 4) -> the PCI doesn't work 

Therefore we have problems! A lot of problems! I had to create a new bootwrapper follows the kernel section. So now, when u-boot attempts to load the image with the 'bootm' command it now extracts the image (the decompression is handled by u-boot) to the $kernel_entry, and then jumps to the bootwrapper start address, located after the kernel image.

The bootwrapper code executes and fixes up a flat device tree for use by the kernel proper, and then jumps the $kernel_entry and the kernel starts up.

before kernel v2.6.25: uboot ----> kernel(@$kernel_entry)
after kernel v2.6.25+: uboot ---> bootwrapper ---> kernel(@kernel_entry)

I had to change Makefiles, linker scripts, Config changes to allow the new Image target, and the boot wrapper also needs to create a correct flat device tree for the target board based off information passed to via the board device structure.


All of the above just to boot the board, and fix the PCI: Crazy!!! Three working weeks gone!

p.s.
and I still have some PCI quirk, but only with PCI_IO, thanks God PCI_MEM is working ..
I am late, will probably try to fix it later. I am not sure I am able to fix. Too damn complex.

[legacy]

Also ...

... different versions of u-boot put the BCSR in different places, and some don't set up the PCI PIC at all, so you have to assume the device tree is sane and update the BRx registers appropriately 

[dgtl]

I have had to migrate to devicetree on a lot of ARM boards, starting over 4 years ago. At first it felt very backwards and difficult. After a while I've started to like it and the new boards using devicetree have much cleaner implementation than old board-init-code based kernels. On none of these boards, there is user-replacable hardware or such, so no autoprobing has been neccessary.
During the migration phase, on devices with older uboot, the devicetree blob was just concatenated to the kernel during kernel build (it can read the blob from there). On newer boards, the uboot handles the dtb by itself. There is support in the uboot for modifying dtb-s dynamically, but I haven't had the need for that.
The devicetree made it simpler to have one single fw image for a lot of different platforms. Earlier it was kind of possible until you ran into something that was configured during build, not runtime. Now I can just build a kernel that has superset of drivers of all boads (that may have different SOC types etc). Ship it with different dtb for each board and I'm done. (For example, one project here runs 5 quite different boards with a single fw image, imx6d/q/dp/qp SOCs).
There has been a steep learning curve, especially due to the lack of documentation as I started migrating to devicetree as ARM started using it and it was new to everybody. Things have changed a lot since, but once in a while I still come by some magic, that takes a lot of digging to work out (Ie some drivers look for nodes with a given string preffix in aliases region for some numbering and collisions happen if something is not as intended). But all in all, things get better all the time.

[legacy]

here I have difficulties with this dts

Code: [Select]

                        reg = <0xeec00000 0x00000008    /* Config space access */
                               0xeed80000 0x00000004    /* IACK */
                               0xeed80000 0x00000004    /* Special cycle */
                               0xef480000 0x00000040>;  /* Internal registers */

                        /*
                         * Outbound ranges, one memory and one IO,
                         * later cannot be changed. Chip supports a second
                         * IO range but we don't use it for now
                         */
                        ranges = <0x02000000 0x00000000 0x80000000   0x80000000 0x00000000 0x20000000
                                  0x01000000 0x00000000 0x00000000   0xb0000000 0x00000000 0x00010000>;


I have already prepared the bootwrapper, but ... the PCI_IO has still problems and I haven't sorted it out yet.

As you can see, uboot load the kernel, dht is built-in in the "early boot".

So, uboot ---> bootwrapper ---> kernel ---> /sbin/init

Code: [Select]

Uboot v1.2

Filename 'kernel-dht'.
Load address: 0x10000
Loading: 5MB, done

# bootm
## Booting image at 00010000 ...
   Image Name:   2.6.39-rc1
   Created:      2017-05-24  14:35:09 UTC
   Image Type:   PowerPC Kernel Image (uncompressed)
   Data Size:    5306108 Bytes =  5.1 MB
   Load Address: 00a00000
   Entry Point:  00a00d78
   Verifying Checksum ... OK

Uboots passes the control to the bootwrapper, which extracts the DeviceTree and passes information to platform_xxx_init, which does some pre-initialization

Code: [Select]

Memory <- <0x0 0x8000000> (128MB)
CPU clock-frequency <- 0xfe49a80 (267MHz)
CPU timebase-frequency <- 0xfe49a80 (267MHz)
/plb: clock-frequency <- 3f926a0 (67MHz)
/plb/opb: clock-frequency <- 1fc9350 (33MHz)
/plb/ebc: clock-frequency <- 1fc9350 (33MHz)
/plb/opb/serial@ef600300: clock-frequency <- a98670 (11MHz)
/plb/opb/serial@ef600400: clock-frequency <- a98670 (11MHz)
ethernet0: local-mac-address <- de:ad:be:ef:de:ad

hAllo, starting: loaded at 0x00a00000 (sp: 0x07fa08c8)
Allocating 0x509480 bytes for kernel ...
gunzipping (0x00000000 <- 0x00a0f000:0x00f0f6fc)...done 0x4f0000 bytes
cmdline:uboot's bootargs overridden
Finalizing device tree... flat tree at 0xf1d300
Using machine description

Then it jumps into the kernel/boot

Code: [Select]

kernel version 2.6.39-flash-eating-bats-II
compiled by root@OrangeCube
compiled with gcc version 4.2.4 (Gentoo 4.2.4-r1 p1.4)
compiled ticket #164 Wed May 24 16:35:01 CEST 2017
bootconsole [udbg0] enabled
setup_arch: bootmem
arch: exit
Zone PFN ranges:
  DMA      0x00000000 -> 0x00003000
  Normal   empty
Movable zone start PFN for each node
early_node_map[1] active PFN ranges
    0: 0x00000000 -> 0x00003000
MMU: Allocated 1088 bytes of context maps for 255 contexts
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 12192
PID hash table entries: 256 (order: -2, 1024 bytes)
Dentry cache hash table entries: 8192 (order: 3, 32768 bytes)
Inode-cache hash table entries: 4096 (order: 2, 16384 bytes)
Memory: 43468k/49152k available (4884k kernel code, 5684k reserved, 172k data, 101k bss, 172k init)
Kernel virtual memory layout:
  * 0xfffdf000..0xfffff000  : fixmap
  * 0xfde00000..0xfe000000  : consistent mem
  * 0xfddfe000..0xfde00000  : early ioremap
  * 0xc4000000..0xfddfe000  : vmalloc & ioremap
SLUB: Genslabs=13, HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
NR_IRQS:512 nr_irqs:512 16
UIC0 (32 IRQ sources) at DCR 0xc0
clocksource: timebase mult[f00625] shift[22] registered
Console: colour dummy device 80x25
pid_max: default: 32768 minimum: 301
Mount-cache hash table entries: 512
devtmpfs: initialized
NET: Registered protocol family 16

 

Look at the PCI section:

Code: [Select]

PCI host bridge /plb/pci@ec000000 (primary) ranges:
 MEM 0x0000000080000000..0x000000009fffffff -> 0x0000000080000000
  IO 0x00000000b0000000..0x00000000b000ffff -> 0x0000000000000000
4xx PCI DMA offset set to 0x00000000
PCI: Probing PCI hardware
PCI: Hiding 4xx host bridge resources 0000:00:00.0
pci 0000:00:03.0: BAR 1: assigned [mem 0x80000000-0x807fffff pref]
pci 0000:00:03.0: BAR 1: set to [mem 0x80000000-0x807fffff pref] (PCI address [0x80000000-0x807fffff])
pci 0000:00:04.0: BAR 5: assigned [mem 0x80800000-0x8081ffff]
pci 0000:00:04.0: BAR 5: set to [mem 0x80800000-0x8081ffff] (PCI address [0x80800000-0x8081ffff])
pci 0000:00:03.0: BAR 6: assigned [mem 0x80820000-0x8082ffff pref]
pci 0000:00:03.0: BAR 0: assigned [mem 0x80830000-0x80833fff]
pci 0000:00:03.0: BAR 0: set to [mem 0x80830000-0x80833fff] (PCI address [0x80830000-0x80833fff])
pci 0000:00:04.0: BAR 6: assigned [mem 0x80834000-0x80837fff pref]
pci 0000:00:02.0: BAR 0: assigned [mem 0x80838000-0x80838fff]
pci 0000:00:02.0: BAR 0: set to [mem 0x80838000-0x80838fff] (PCI address [0x80838000-0x80838fff])
pci 0000:00:02.1: BAR 0: assigned [mem 0x80839000-0x808390ff]
pci 0000:00:02.1: BAR 0: set to [mem 0x80839000-0x808390ff] (PCI address [0x80839000-0x808390ff])
pci 0000:00:04.0: BAR 4: assigned [io  0x1000-0x103f]
pci 0000:00:04.0: BAR 4: set to [io  0x1000-0x103f] (PCI address [0x1000-0x103f])

With Bar4 (PCI_IO) something went wrong

Code: [Select]

PROMISE PDC20265 0000:00:04.0: device not available (can't reserve [io  0x01f0-0x01f7])
PROMISE PDC20265: probe of 0000:00:04.0 failed with error -22

Indeed, it can't reserve IO xxxx

There is a conflict, and can't sort out how to fix it.

Any idea  ?

[dgtl]

Sorry, haven't had to set up PCI like this.
Why aren't you doing your work in u-boot, why is that wrapper needed? Hasn't anyone already done the same work in the u-boot tree, so just updating u-boot would get you all the changes that are needed? Adding a separate shim inbetween uboot and kernel is imho just looking for trouble on both sides, every change on both side will break it. I would first update to the latest uboot and then start working on the uboot from there.

[legacy]

Why aren't you doing your work in u-boot

Because I can't touch the firmware, and DENX doesn't support our board in their modern version of uboot.

why is that wrapper needed?

Because we must use cuImage (aka uImage legacy). Kernel > 2.6.25 doesn't bootstrap with New-uImage, because in the linux team they changed the early boostrap code inside the kernel.

Hasn't anyone already done the same work in the u-boot tree

Not for the board we need to support.

just updating u-boot would get you all the changes that are needed?

Uboot v2017 doesn't bootstrap at all on the prototype we have in our laboratory, and it seems the problem goes deeper, the original code has changed a lot (since 2007, 10 years ago), so a lot of new work is required to understand WTF is broken in modern versions.

From my point of view: DENX's sources are a mess!
From my boss's point of view: fixing new versions is ... a cost.
From our customer's point of view ... it's a noGo!

Adding a separate shim inbetween uboot and kernel is imho just looking for trouble on both sides, every change on both side will break it. I would first update to the latest uboot and then start working on the uboot from there.

I know, but we can't update uboot.

There are platforms where the firmware can't be updated so easily, e.g. the boards installed on our customer's cruise ships. It's physically impossible. We can send a new kernel + rootfs through a (ssh) radio link, files will be uploaded into a tftpboot server, and reloaded during system reset, but we can't remotely reflash the firmware. It's a noGo.

To physically update the firmware we have to physically access the board, and it's a cost.

From my point of view it's silly! Everything form kernel 2.4 to 2.6.25 worked without the need to change the firmware!

[dgtl]

I understand your reasoning. In that case, the workaround is appropriate (although not a nice fix).

As I'm working for a company that does the both the hw and fw, we must customize the u-boot support anyway ourselves too, and that support can be ported to newer versions as needed. Also during my experience of 10+ years, almost every project needs a couple of bootloader updates, so it is always planned into every product (mostly these updates are needed for the first manufacturing batches of couple of hundred devices as some difficult to predict tolerances creep out). Later things usually stabilize, but some issues sometimes still happen. (The worst has been a case where the soc eval board had NAND flash ECC settings messed up; we based our device bootloader on that and did not find that issue until hundreds of devices live... The devices started failing as soon as the NAND flashes started getting first bad bits, in about half a year or so. Re-flashing a bootloader in RAW mode using software-calculated OOB ECC blocks using OTA update on production devices was very risky, but not doing it would have caused a massive outage in a couple of weeks, so remote bootloader update saved the day)
As usually the bootloader update can not be done in a safe way, this must be only as needed, no just-in-case updates with same version or any nice-to-have updates. Our devices fw update packages always contain the bootloader blob too, but it is always checked if that needs to be flashed before doing the programming. The update is done on linux-based devices always after clean boot without anything started and the whole system running from RAM, no flash access uncertainty during the bootloader update.

[legacy]

Also, as far as I see from sources, it seems ARM's code is cleaner  than PPC's about PCI