Author Topic: I am lost with the PCI bar size  (Read 1910 times)

0 Members and 1 Guest are viewing this topic.

Online DiTBhoTopic starter

  • Super Contributor
  • ***
  • Posts: 3894
  • Country: gb
I am lost with the PCI bar size
« on: February 05, 2023, 09:58:17 pm »
The Intel' PCI specs says you can discover the size of a BAR by writing all ones to it and reading it back, which tells us how many bits of the BAR are hard-wired to zero.

Right?

So when the firmware does it and you see size=0 ... is it likely a hardware defect? of the PCI card? of the PCI controller?

And, can it be safely ignored?  :-//
The opposite of courage is not cowardice, it is conformity. Even a dead fish can go with the flow
 

Online magic

  • Super Contributor
  • ***
  • Posts: 6749
  • Country: pl
Re: I am lost with the PCI bar size
« Reply #1 on: February 06, 2023, 11:25:57 am »
Prolly one of:

1. everything reads as zeros - the card is badly inserted, controller bug, software bug, etc.
2. you are again dealing with some proprietary obsolete shit which was never meant to work with anything but its corresponding proprietary obsolete shit drivers.
 
The following users thanked this post: DiTBho

Online Nominal Animal

  • Super Contributor
  • ***
  • Posts: 6227
  • Country: fi
    • My home page and email address
Re: I am lost with the PCI bar size
« Reply #2 on: February 06, 2023, 06:08:35 pm »
If this is the Commel MP-6421 module (1106:3249), PCI SATA+CF, based on VIA VT6421A (datasheet), it behaves per PCI 2.2, and runs (both PCI and SATA) at 3.3V.  It uses a 128-byte (64-word) SPI EEPROM, and to configure PCI properly, the PCI Base and Limit registers at offsets 0x70 and 0x74 need to be programmed, as they default to – no surprises here – zero.

Section 6.2.5 of the PCI Local Bus Specification Revision 2.2 describes how the BIOS (power-up software) should set up the BAR(s).  Quite likely your BIOS or its equivalent does not do that, so you'll have to do it in software.  In Linux, that would be very early, between base PCI init (because you want PCI bridges to be initialized) but before device probing; so most likely in arch-dependent code.

If you run Linux on a machine that does have hardware 3.3V PCI support, but no BIOS that initializes PCI devices correctly, and have the MP-6421 module installed, then the VIA SATA driver should error out in drivers/ata/sata_via.c:svia_init_one() with error "invalid PCI BAR # (sz 0, val 0)", or similar (if not earlier).  Again, the fix is to set those registers somewhere very early in PCI init, most likely in arch-specific code.

However, I could be completely wrong here.  I do not have much experience with the hardware/pre-driver side of PCI (or PCIe) devices, so I'm basing this analysis partly on guesses (based on the MP-6421 we discussed in PMs), and the rest on what kind of scenario could lead to PCI base and limit being detected as zeroes in the Linux driver.
 
The following users thanked this post: DiTBho

Online DiTBhoTopic starter

  • Super Contributor
  • ***
  • Posts: 3894
  • Country: gb
Re: I am lost with the PCI bar size
« Reply #3 on: February 07, 2023, 12:02:56 am »
the BIOS (power-up software) should set up the BAR(s)

yeah, that kind of hack, but at the moment I'm fighting with a close-source firmware called "RouterBOOT v2.8" since I don't know how it initializes the PCI, and it looks * whatever it's doing * it's doing it wrong.

I did something similar for a PowerPC embedded card, but I was able to replace IBM's proprietary firmware with u-boot, so I was able to write all the pre-initialization of the specific device.

(I prefer to have that part inside the bootloader, so I don't have to fight with the Linux Kernel when I have to "move" my code form kernel v4.* to kernel v6.*)

Anyway I'm not a PCI expert, what I wrote was a simple pre-initialization for a pATA controller that bootstraps in "legacy-LBA" mode and needs a couple of commands to switch to "PCI" mode.

I mean, I know how to write machine-specific code in the first section of the Linux kernel, but I still don't know exactly what I have to write, and I wish I could reprogram the FLASH with u-boot instead of RouterBOOT

All of this, in order to add a bloody sATA disk to a router :-//
The opposite of courage is not cowardice, it is conformity. Even a dead fish can go with the flow
 

Online Nominal Animal

  • Super Contributor
  • ***
  • Posts: 6227
  • Country: fi
    • My home page and email address
Re: I am lost with the PCI bar size
« Reply #4 on: February 07, 2023, 04:48:04 am »
at the moment I'm fighting with a close-source firmware called "RouterBOOT v2.8" since I don't know how it initializes the PCI, and it looks * whatever it's doing * it's doing it wrong.
Yep, that is a real pain in the butt.  It should be something like coreboot src/device/pci_device.c:pci_get_resource(), but isn't.   >:(

Latest working RouterOS version for RB532A is 6.32.4, so you might wish to download that and update RouterBOOT to that version.

Looking at OpenWRT bootlog on RB532A (kernel 3.10.34):
 - pci 0000:00:00.0 is the IDT PCI root
 - pci 0000:00:02.0 is VIA Rhine VT6106S Ethernet
 - pci 0000:00:03.0 is VIA Rhine VT6106S Ethernet
 - pci 0000:00:04.0 is Lite-On WLAN card (mini-PCI)
 - pci 0000:00:05.0 is Lite-On WLAN card (mini-PCI)
Only the IO BARs get assigned, none of the memory BARs do.  (The third Ethernet port "korina", PCI 011d:0214, is integrated on the SoC.)

The SoC itself is IDT (Renesas) 79RC32H435 (Mouser datasheet).  It seems to have a perfectly functional 32-bit PCI 2.2 bus, but the datasheet mentions its local memory/IO controller has a 26-bit address bus.

Augh.  I only realized when writing this post, that arch/mips/pci/*rc32434.c also belong to the RB532A kernels, and it already has PCI fixups for the korina ethernet port, writing the korina configuration registers (similar to PCI V2X registers on VT6421A)!  So, essentially, you need to add your own PCI fixups to arch/mips/pci/pci-rc32434.c:rc32434_pcibridge_init(), similar to the for loop that copies the 25 korina_cnfg_regs[] entries to the corresponding PCI addresses.  I am not 100% sure, but looking at the drivers/ata/sata_via.c for VT6421A, it looks like it only needs IO bars for the driver to work.

In other words, this board does not have a "BIOS" to properly initialize anything PCI, and it has to be done in the Linux kernel (or whatever you want to run).

(It looks like VT6421A is able to store its configuration defaults (including PCI V2X registers) in EEPROM, up to 63 8-bit changes to the defaults, in words/byte pairs where upper byte indicates the register offset, and lower byte the new value.  If the EEPROM includes pairs 0x70xx and 0x71xx, it defines the PCI V2X base address; if the EEPROM includes pairs 0x72xx and 0x73xx, it defines the limit, and so on.  See the VT6421A datasheet for details.)
 
The following users thanked this post: DiTBho

Online DiTBhoTopic starter

  • Super Contributor
  • ***
  • Posts: 3894
  • Country: gb
Re: I am lost with the PCI bar size
« Reply #5 on: February 07, 2023, 09:54:07 am »
Latest working RouterOS version for RB532A is 6.32.4, so you might wish to download that and update RouterBOOT to that version.

umm, "RouterBoot" is the "bootloader", while "RouterOS" is a ... dunno-what-it-is you need to buy ... perhaps a license to use it? it's a "close-source OS" of which I don't even have the admin passwd, all I know is that commercial binary blob goes into the onboard flash, and here we have an other problem with the bootloader:

Code: [Select]
trying bootp protocol..... OK
Got IP address: 192.168.1.4
resolved mac address 00:02:8A:26:B2:1C
transfer started .......................................................... transfer ok, time=2.80s
setting up elf image... kernel out of range
kernel loading failed

boooooom, "kernel out of range", means you cannot bootstrap a kernel bigger than 7700000 byte  :o :o :o :o

  • nand-boot
  • cf-boot
  • tftp-boot
three methods to boot, all fail the same way

I have already updated my builder accordingly
Code: [Select]
/src/machine/mips/rb532/kernel/5.4.128 # ./mybuild-2023 2

kernel-5.4.128(mips/rb532)
toolchain(mipsel-unknown-linux-gnu:2.34/9.3.0)
app_name       = kernel-5.4.128-32bit-rb532
machine_note   = revB-experimental-ath9kwifi-korina-rhine-pciserial
qualified_host = { macmini-intel, LaptopT220 }
-----------------------------------------------
gadget_to_do(build) ... overlay/build
gadget_to_do(clean) ... kernel/clean
app_machine=rb532 ... [MIPS32R2/LE]
building kernel ... success
-----------------------------------------------
app_size    =7840728 byte
app_size_max=7700000 byte
checking size ... panic
    module=profile/do/kernel/clean
    reason="kernel size too big for the firmware"


kernel out of range, it's tooooooooooooo big for the firmware  :-//


So, as emergency-plan, I am developing a kind of elf application, just to
- initialize the CPU (minimal)
- initialize tlb and ram (with a window of 16Mbyte)
- initialize the eth0 (korina)
- load a kernel from udp/ip into ram
- flush cache and prepare tlb
- jump into kernel_start

That crazy stuff is less than 1Mbyte and it will go into the flash "OS" partition

     flash partition#0 RouterBOOT (don't touch it, or you will brick the router)
     flash partition#1 RouterOS <------------ my app will go here

@reset -> RouterBOOT
RouterBOOT -> load app.elf from Flash
app.elf (my bootloader) -> load kernel.elf from net, then jump into $start

___ OMG, why life is so complex?

I would most likely throw the router away and buy a modern one ...
... anyway, this is the plan for 2023 ;D
« Last Edit: February 07, 2023, 06:28:14 pm by DiTBho »
The opposite of courage is not cowardice, it is conformity. Even a dead fish can go with the flow
 

Online DiTBhoTopic starter

  • Super Contributor
  • ***
  • Posts: 3894
  • Country: gb
Re: I am lost with the PCI bar size
« Reply #6 on: February 07, 2023, 09:57:59 am »
p.s.
I am back to kernel-v4.4.197.
yup, older, but it's smaller and more stable  :o :o :o
The opposite of courage is not cowardice, it is conformity. Even a dead fish can go with the flow
 

Online DiTBhoTopic starter

  • Super Contributor
  • ***
  • Posts: 3894
  • Country: gb
Re: I am lost with the PCI bar size
« Reply #7 on: February 07, 2023, 10:14:46 am »
kernel-v4.4.197
Code: [Select]
PCI: Initializing PCI
PCI host bridge to bus 0000:00
pci_bus 0000:00: root bus resource~ [mem 0x50000000-0x5fffffff]
pci_bus 0000:00: root bus resource~ [io  0x18800000-0x188fffff]
pci_bus 0000:00: root bus resource~ [??? 0x00000000 flags 0x0]
pci_bus 0000:00: No busn resource found for root bus, will use [bus 00-ff]
pci 0000:00:00.0: reg0x14'size is zero, drivers/pci/probe
pci 0000:00:00.0: reg0x18'size is zero, drivers/pci/probe
pci 0000:00:04.0: BAR 0: assigned [mem 0x50000000-0x5001ffff]
pci 0000:00:04.0: BAR 1: assigned [mem 0x50020000-0x5003ffff]
pci 0000:00:04.0: BAR 6: assigned [mem 0x50040000-0x5005ffff pref]
pci 0000:00:02.0: BAR 0: assigned [io  0x18800000-0x188000ff]
pci 0000:00:02.0: BAR 1: assigned [mem 0x50060000-0x500600ff]
pci 0000:00:03.0: BAR 0: assigned [io  0x18800400-0x188004ff]
pci 0000:00:03.0: BAR 1: assigned [mem 0x50060100-0x500601ff]
pci 0000:00:04.0: BAR 2: assigned [io  0x18800800-0x1880083f]
(stripped out to minimal)
The opposite of courage is not cowardice, it is conformity. Even a dead fish can go with the flow
 

Online DiTBhoTopic starter

  • Super Contributor
  • ***
  • Posts: 3894
  • Country: gb
Re: I am lost with the PCI bar size
« Reply #8 on: February 07, 2023, 12:28:35 pm »
Code: [Select]
rb532 / # lspci
00:00.0 Non-VGA unclassified device: Integrated Device Technology, Inc. Device 0000
00:02.0 Ethernet controller: VIA Technologies, Inc. VT6105/VT6106S [Rhine-III] (rev 86)
00:03.0 Ethernet controller: VIA Technologies, Inc. VT6105/VT6106S [Rhine-III] (rev 86)
00:04.0 Ethernet controller: Intel Corporation 82541GI Gigabit Ethernet Controller (rev 05)

00:04.0 is a miniPCI i82541 single-channel ethernet card, using both PCI_IO and PCI_mem  :o :o :o

Code: [Select]
00:04.0 Ethernet controller: Intel Corporation 82541GI Gigabit Ethernet Controller (rev 05)
        Subsystem: Intel Corporation PRO/1000 MT Network Connection
        Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 142
        Memory at 50000000 (32-bit, non-prefetchable) [size=128K]
        Memory at 50020000 (32-bit, non-prefetchable) [size=128K]
        I/O ports at 18800800 [size=64]
        [virtual] Expansion ROM at 50040000 [disabled] [size=128K]
        Capabilities: [dc] Power Management version 2
        Capabilities: [e4] PCI-X non-bridge device
The opposite of courage is not cowardice, it is conformity. Even a dead fish can go with the flow
 

Online Nominal Animal

  • Super Contributor
  • ***
  • Posts: 6227
  • Country: fi
    • My home page and email address
Re: I am lost with the PCI bar size
« Reply #9 on: February 07, 2023, 05:01:47 pm »
Latest working RouterOS version for RB532A is 6.32.4, so you might wish to download that and update RouterBOOT to that version.
umm, "RouterBoot" is the "bootloader", while "RouterOS" is a ... dunno-what-it-is you need to buy
Yes, I know.  A RouterOS license is included with the RB532A board.  The bootloader is paired with the RouterOS.
Again, the latest version supported without issues by RB532A is 6.32.4, so I don't believe it would cost you anything (except time and internet connection).

Essentially, you'd have to install routeros-mipsle-6.32.4 for the duration of upgrading the bootloader, and then reinstall your preferred OS.  (The documentation says there is a way to upgrade only the bootloader via a serial cable –– "Note: RouterOS cannot be upgraded through serial cable. Using this method only RouterBOOT can be upgraded. " –– but the link does not exist in their Wiki.  Might be worth investigating, though; the bootloader serial is 38400,8,N,1 as you know, and the option in bootloader config is 'g'.  The bootloader should be able to boot a Linux kernel over Ethernet (TFTP), too.)

I am not sure if it would help with the issues you have, though.  It's just what I'd do (update the bootloader to the latest good version) to eliminate possible sources of problems, one by one.

I would most likely throw the router away and buy a modern one ...
Now you know first-hand why I claim software engineering hasn't really progressed in the last two or three decades: we have just more code now, requiring more resources than before.

As to the elf application, you could also use a minimal kernel (and minimal userspace on initrd or mounted read-only), and use kexec to load and switch to the actual kernel.

Code: [Select]
PCI: Initializing PCI
PCI host bridge to bus 0000:00
pci_bus 0000:00: root bus resource~ [mem 0x50000000-0x5fffffff]
pci_bus 0000:00: root bus resource~ [io  0x18800000-0x188fffff]
pci_bus 0000:00: root bus resource~ [??? 0x00000000 flags 0x0]
pci_bus 0000:00: No busn resource found for root bus, will use [bus 00-ff]
pci 0000:00:00.0: reg0x14'size is zero, drivers/pci/probe
pci 0000:00:00.0: reg0x18'size is zero, drivers/pci/probe
pci 0000:00:04.0: BAR 0: assigned [mem 0x50000000-0x5001ffff]
pci 0000:00:04.0: BAR 1: assigned [mem 0x50020000-0x5003ffff]
pci 0000:00:04.0: BAR 6: assigned [mem 0x50040000-0x5005ffff pref]
pci 0000:00:02.0: BAR 0: assigned [io  0x18800000-0x188000ff]
pci 0000:00:02.0: BAR 1: assigned [mem 0x50060000-0x500600ff]
pci 0000:00:03.0: BAR 0: assigned [io  0x18800400-0x188004ff]
pci 0000:00:03.0: BAR 1: assigned [mem 0x50060100-0x500601ff]
pci 0000:00:04.0: BAR 2: assigned [io  0x18800800-0x1880083f]
Aside from the harmless "pci_bus 0000:00: root bus resource~ [??? 0x00000000 flags 0x0]", it seems to me the hardware is now detected and mapped correctly, and you have solved all the PCI issues you had.
 
The following users thanked this post: DiTBho

Online DiTBhoTopic starter

  • Super Contributor
  • ***
  • Posts: 3894
  • Country: gb
Re: I am lost with the PCI bar size
« Reply #10 on: February 07, 2023, 06:17:49 pm »
nah, it works with the miniPCI 82541GI Gigabit Ethernet card, but not with the  VIA-VT6421A card nor with the SIL3112 card.

Unlucky with any tested sATA card  :-//

These two cause such strange behavior during PCI probing that you need to do a hardware reset (or unplug the power supply) to resume.

I would like to find and buy an EHCI miniPCI with a NEC chip. I'm 99% sure it will work without any problems with the rb532A!
The opposite of courage is not cowardice, it is conformity. Even a dead fish can go with the flow
 

Online DiTBhoTopic starter

  • Super Contributor
  • ***
  • Posts: 3894
  • Country: gb
Re: I am lost with the PCI bar size
« Reply #11 on: February 07, 2023, 07:26:11 pm »
Despite what's written in the doc, my board is brand new but doesn't have RooterOS installed.
The FLASH is empty.

I have already downloaded the "routeros-mipsle-6.32.4.npk" file, but not yet understood how to use it.
I tried a simple "firmware upgrade via ethernet", the tftpboot server sends the file to the router, but it doesn't work because the router complains the file-ID doesn't match something.
Code: [Select]
trying bootp protocol... OK
Got IP address: 192.168.1.41
resolved mac address 00:02:8A:26:B2:1C
transfer started ......................................................... transfer ok, time=2.70s
invalid upgrade file id

The other methods require RooterOS installed and running.

So, I will contact Mikrotik, and ask what to do, and the change log.

Meanwhile, found and bought a new rb532A board, this time it comes with RouterBOOT-v2.9.

We make progress, from RouterBOOT-v2.8 to v2.9  :D :D :D
The opposite of courage is not cowardice, it is conformity. Even a dead fish can go with the flow
 

Online Nominal Animal

  • Super Contributor
  • ***
  • Posts: 6227
  • Country: fi
    • My home page and email address
Re: I am lost with the PCI bar size
« Reply #12 on: February 07, 2023, 08:49:14 pm »
nah, it works with the miniPCI 82541GI Gigabit Ethernet card, but not with the  VIA-VT6421A card nor with the SIL3112 card.
Right; so we know the RB532A works, just not with the two SATA bridge cards you have.

In my opinion (which may not be worth that much!), that indicates that the VT6421A and SIL3112 cards require early PCI initialization to work.  Both support EEPROM to initialize configuration register values.  If you can remove the EEPROM chip, the two datasheet has enough information for us to (read and) reprogram the contents so that the card itself initializes its PCI bus settings correctly.  Right now, they're probably preset to some x86-compatible settings, which may not work for the RB532A.



Would you be willing to try a patch to arch/mips/pci/pci-rc32434.c, to see if the VT6421A or SIL3112 could be initialized to get one of them working?

Before that, right now, I would suggest adding two (early?) PCI fixup handlers in arch/mips/pci/fixup-rc32434.c, that do the same thing: dump the PCI configuration space register values,
Code: [Select]
static void dump_sil3112a(struct pci_dev *pdev)
{
    u32 v;
    int i;
    for (i = 0; i <= 0xC4; i += 4) {
        pci_read_config_dword(pdev, i, &v);
        printk(KERN_INFO "SIL3112A [%x] = %x\n", i, v);
    }
}

DECLARE_PCI_FIXUP_HEADER(0x1095, 0x3112, dump_sil3112a);

static void dump_vt6421a(struct pci_dev *pdev)
{
    u32 v;
    int i;
    for (i = 0; i <= 0xD2; i += 4) {
        pci_read_config_dword(pdev, i, &v);
        printk(KERN_INFO "VT6421A [%x] = %x\n", i, v);
    }
}

DECLARE_PCI_FIXUP_HEADER(0x1106, 0x3249, dump_vt6421a);
If the kernel locks up before you get the output, switch both to DECLARE_PCI_FIXUP_EARLY –– although that might interfere with printk().

For the SIL3112A, 10,14,18,1C,20 are the five IO BARs (and must have LSB set); 24 is the mem BAR the linux driver uses and should have low 10 bits clear; and 30 is the ROM address, and should have LSB clear (disabled).

For the VT6421A, 60 is boot ROM address, 70 shows PCI V2X base and limit, 88 primary SG base address, 8C secondary SG base address.

All the rest are useful too, because they can be compared to the default values shown in the datasheets.  Knowing the values of all these, since the Linux kernel drivers only use a subset of the features so the rest can/should be left to chip defaults, a PCI fixup/early fixup can use a very similar loop to reconfigure the cards.

Another approach is to move from PCI fixups to arch/mips/pci/pci-rc32434.c:rc32434_pcibridge_init(), and use a loop to detect and then initialize the cards' PCI registers as soon as the PCI bridge itself is initialized, just like the korina PCI config registers for the SoC built-in ethernet.
I'd prefer the fixups myself, though.  But, without knowing what the cards' PCI registers, without dumping the register contents as in the above addition, it's pretty much impossible to do because we don't yet know which registers get initialized to invalid values.  (I suspect the ROM apertures, myself.  For RB532A they most likely need to be disabled.)
« Last Edit: February 07, 2023, 08:55:30 pm by Nominal Animal »
 
The following users thanked this post: DiTBho

Online DiTBhoTopic starter

  • Super Contributor
  • ***
  • Posts: 3894
  • Country: gb
Re: I am lost with the PCI bar size
« Reply #13 on: February 08, 2023, 07:55:07 am »
cards require early PCI initialization to work.  Both support EEPROM to initialize configuration register values

Yup, it's exactly what I did with u-boot on the PowerPC-embedded SBC to make the PCI-pATA card work.

That bloody pATA controller boots in "legacy mode", which is like "ISA16" bus, which is a common IBM-PC design choice, crappy design because not supported in HPPA workstations and PowerMacs, which expect a PCI-pATA-devide working like a PCI-device/class-storage.

"legacy mode" means the card doesn't respond to the PCI-probing and instead it assumes it can operate with a predefined setup that is only OK on PCs, and what if that setup, in a non-PC machine, conflicts with other devices? ..

I spent two years to fix it, because it was not documented.
The opposite of courage is not cowardice, it is conformity. Even a dead fish can go with the flow
 

Online DiTBhoTopic starter

  • Super Contributor
  • ***
  • Posts: 3894
  • Country: gb
Re: I am lost with the PCI bar size
« Reply #14 on: February 08, 2023, 08:01:22 am »
I suspect the ROM apertures, myself.  For RB532A they most likely need to be disabled.

I suspect too. Bought a second SIL card because (it's cheap (5UKP) and) you can easily remove the Flash and unsolder the serial EEPROM.

On the VIA card the flash is high-density. I will try to unsoder the /CE pin, it should disable the chip  :-//
The opposite of courage is not cowardice, it is conformity. Even a dead fish can go with the flow
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf