Bit-Brick K1

[SiliconWizard]

I've been considering working on baremetal dev on a SpacemiT K1 SoC, and the cheap candidate as a dev board was the Orange Pi RV2 (there are alternatives that didn't seem to bring a lot of benefits for that).

But I found this one, which is fully open-source: Bit-Brick K1
https://www.bit-brick.com/k1/

Available on Aliex for sale, a tad more expensive than the Pi RV2 but not by much. While it only has 1 Ethernet port instead of 2, it uses the more familiar RTL8211F PHY. And it has a lot more GPIOs broken out.

Wondering what you would recommend for cooling a SpacemiT K1? I suppose passive cooling with a heatsink should be fine.

[DiTBho]

I've been considering working on baremetal dev on a SpacemiT K1 SoC

Why not GNU/Linux?
Are you going to also develop any network driver?
I had to develop baremetal dev on RB532 in order to make a bootloader (because the firmware is too limited) ... and it was very very complexy.
Plus, I had to deal directly with PCI.

Wondering what you would recommend for cooling a SpacemiT K1? I suppose passive cooling with a heatsink should be fine.

with a big and massive heatsink.
Otherwise add a cooling fan.

Don't make the same mistake made by FriendlyArm with their Neo boards where the passive cooling was enough *ONLY* if the CPUs are <50% load.
I had to hack the firmware, reduce the clock frequency from 1.1Ghz to 600Mhz.

[SiliconWizard]

I did this with the CV1800B and I expect it not to be a lot more difficult with the SpacemiT K1, although obviously it has more features and I'm not going to implement full support.
My goal is to implement network support, yes - did that recently on the CH32V307 and again, while the MAC is not the same, I expect it to be relatively similar.
Probably not going to implement PCIe support, at least for a while. That may be limiting if I want to be able to use SSD storage, though, so I'll see how difficult it is to implement minimal PCIe + NVMe support. Probably not a picnic.

As to heatsinking, there's unfortunately no real documentation on what to use. Most SBCs around this SoC come without heatsinks (as usual), except the LicheePi 3A, which comes with a small heatsink with a fan on top.
The datasheet doesn't tremendously help - they state a max "digital" current of 10 A, but that's probably greatly exaggerated. Given that most of it is drawn at the core voltage (which is roughly between 0.6 V and 1 V), one can expect 10 W max in probably the worst case. I've seen power consumption measurements taken with boards using this SoC, which usually max out at around 6-7 W (for the whole board). The SoC itself probably doesn't exceed half that. So, that shouldn't require a very large heatsink. I've ordered small heatsinks made for the Orange Pi RV2, and I'll see how that goes.

For those interested, the documentation for the K1 is there:
https://developer.spacemit.com/documentation?token=DBd4wvqoqi2fiqkiERTcbEDknBh
https://developer.spacemit.com/documentation?token=YEAtw51p6ixJS5kQHsFclqT0naf

It can be browsed online or downloaded as PDFs. (From what I can tell, it must have been made using asciidoc.)
The docs are not too bad, although not very detailed - the manual is "only" 791 pages, which is very little for such a SoC, but at least all registers seem to be documented.
I couldn't find a doc specific to the X60 core that this SoC uses though, so while it's compliant with extensions that are listed, it does implement some custom CSRs that are not documented. I suppose one would need to have a NDA with SpacemiT. I'll ask them.

The plan is to boot this way:

BootROM -> FSBL -> OpenSBI -> My firmware

Pretty much the same approach I used for the CV1800B. I might get rid of the OpenSBI step at some point, as the low-level initialization, such as DDR RAM controller init and training, is done by the FSBL (TBC here).

The FSBL can be found in U-Boot's source code (although it's technically not a direct part of U-Boot and I don't intend to use U-Boot after OpenSBI).

The U-Boot source code adapted to the K1 (specifically for this board) can be found here: https://github.com/bit-brick/k1-uboot-2022.10.git
And for OpenSBI, the best bet seem to be this repo: https://gitee.com/bianbu-linux/opensbi.git

Building both was relatively straightforward.

I should receive the board in about a week now, so I'll see how that goes. The perspective of having a platform with a 8-core RV64 SoC at 1.6 GHz, with RVV1.0, a lot of peripherals and several GB of RAM, that could be programmed baremetal opens quite a few possibilities. The CV1800B was already a pretty interesting step, but this SoC is a whole other world yet.

[DiTBho]

My goal is to implement network support, yes - did that recently on the CH32V307 and again, while the MAC is not the same, I expect it to be relatively similar.

here you were lucky, on the rb532a router the network part depends heavily on the PCI because it is a PCI peripheral external to the SoC.

[SiliconWizard]

Ah yes, fortunately on this SoC, everything is integrated - USB, Ethernet MACs (2), SDIO, peripherals such as UART, SPI, I2C, ... even a GPU, all internal and accessible via their own registers.

For the record, this SoC claims 2 TOPS  "AI" acceleration, but contrary to some others, it doesn't actually have a dedicated NPU. It implements a specific AI RISC-V extension (so a bunch of instructions) in the first 4 of the 8 cores. So I suppose the "2 TOPS" are estimated based on 100% dedicated use of those 4 cores for this purpose. It's an interesting approach, as it probably lowers cost significantly, and makes it possible to use those instructions along with all other instructions and thus more easily use these for other purposes than running CNNs. Could be interesting. Those 4 cores have access to a 512 KB tightly-coupled SRAM, which is "dedicated" to this AI extension, but at first sight, I don't see a reason why it couldn't be used as regular memory for any other purpose, so there you have 512 KB of TCM. Should be cool for putting time-critical code as long as it can run from there - I'll have to test that.

As I said, PCIe would be needed only to access external storage on PCIe SSDs (and possibly other purposes for handling PCIe cards). I don't know yet how hard it is to implement. Otherwise, it supports SD and eMMC, so one can use that as storage, which is significantly easier.

As a "desktop" CPU, tests show that it gives rather unimpressive performance (at least compared to what we expect these days), which makes sense, but as a kind of "super-application processor", it looks like a killer.

[DiTBho]

For the record, this SoC claims 2 TOPS  "AI" acceleration, but contrary to some others, it doesn't actually have a dedicated NPU. It implements a specific AI RISC-V extension (so a bunch of instructions) in the first 4 of the 8 cores.

Can I ask which instructions?
I do find it *very* interesting

[SiliconWizard]

Their extension is called IME and the spec can be found here: https://github.com/spacemit-com/riscv-ime-extension-spec
It piggybacks on the vector extension.

As its target is "AI", it's limited to 8-bit and 16-bit operands (integers or FP), but if you have some use for that, it gives you dot-product matrix multiply-accumulate instructions.

[SiliconWizard]

I received the board and did some testing.

I ordered the 8GB RAM version, and while it doesn't appear on the product page (that I could see at least), the board came with a 64 GB eMMC on board. Pretty cool. It was pre-flashed with Bianbu Linux 2.2, so I could quickly issue some preliminary tests.

Code: [Select]

        #####           root@k1
       #######          -------
       ##O#O##          OS: Bianbu 2.2.1 riscv64
       #######          Host: spacemit k1-x bit-brick board
     ###########        Kernel: 6.6.63
    #############       Uptime: 19 mins
   ###############      Packages: 2214 (dpkg)
   ################     Shell: bash 5.2.21
  #################     Terminal: /dev/pts/0
#####################   CPU: Spacemit X60 (8) @ 1.600GHz
#####################   Memory: 545MiB / 7834MiB
  #################

I tested Bruce's primes benchmark, but am a bit disappointed with the results: 31.2 billion cycles. In another thread, Bruce did test the LicheePi 3A (same SoC) and got ~23 billion cycles. Not sure what could explain the difference.
The cores appear to all run at 1.6 GHz. (Mentioning it, as I used 'rdtime' to estimate CPU cycles, as 'rdcycle' is not available in user mode.)

Code: [Select]

cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_cur_freq
1600000
1600000
1600000
1600000
1600000
1600000
1600000
1600000

Any idea?

[brucehoult]

I just booted up the LicheePi 3A and ran the existing primes binary and got:

Code: [Select]

bruce@lpi3a:~$ ./primes
Starting run
3713160 primes found in 14673 ms
214 bytes of code in countPrimes()
bruce@lpi3a:~$ uname -a
Linux lpi3a 6.6.36 #2.0.4.2 SMP PREEMPT Thu Dec  5 15:02:13 UTC 2024 riscv64 riscv64 riscv64 GNU/Linux
bruce@lpi3a:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Bianbu
Description: Bianbu 2.0.4
Release: 2.0.4
Codename: noble

That's even slightly better than in the primes.txt file.

Compiling a fresh one with the preinstalled gcc (Ubuntu 13.2.0-23ubuntu4bb2) 13.2.0 with only "-O" gave the same size binary and the same results, within the usual experimental error .. 14754 first time, 14668 2nd.

Looking earlier in the thread, which I had somehow missed, the Orange Pi RV2 is -- according to SpacemiT staff -- not in fact the same SoC, but it is very closely related.

[SiliconWizard]

I am testing not on an Orange Pi RV2 but on a Bit-Brick K1 which has a genuine K1 (as far as I can tell).
Using time gave ~ 19s, consistent with the timing I got using rdtime (so that confirms it's apparently not just a clock issue).

Here is the disassembly of the test code:

Code: [Select]

        .file   "primes.c"
        .option nopic
        .attribute arch, "rv64i2p1_m2p0_a2p1_f2p2_d2p2_c2p0_zicbom1p0_zicbop1p0_zicboz1p0_zicond_zicsr2p0_zifencei2p0_zba1p0_zbb1p0_zbc1p0_zbs1p0"
        .attribute unaligned_access, 1
        .attribute stack_align, 16
        .text
        .section        .rodata.str1.8,"aMS",@progbits,1
        .align  3
.LC0:
        .string "SpacemiT X60: countPrimes() ..."
        .align  3
.LC1:
        .string "nPrimes = %d\n"
        .align  3
.LC2:
        .string "Clock cycles = %g\n"
        .section        .text.startup,"ax",@progbits
        .align  1
        .globl  main
        .type   main, [member=46715]function[/member]
main:
.LFB55:
        .cfi_startproc
        addi    sp,sp,-32
        .cfi_def_cfa_offset 32
        lui     a0,%hi(.LC0)
        addi    a0,a0,%lo(.LC0)
        sd      ra,24(sp)
        sd      s0,16(sp)
        sd      s1,8(sp)
        .cfi_offset 1, -8
        .cfi_offset 8, -16
        .cfi_offset 9, -24
        call    puts
#APP
# 19 "primes.c" 1
        rdtime s1
# 0 "" 2
#NO_APP
        lui     t0,%hi(nSieve)
        lw      a5,%lo(nSieve)(t0)
        li      a4,2
        lui     t5,%hi(primes)
        lui     t4,%hi(sieve)
        addi    t5,t5,%lo(primes)
        addi    t4,t4,%lo(sieve)
        addiw   t3,a5,1
        sw      a4,0(t5)
        li      a4,4
        sw      t3,%lo(nSieve)(t0)
        li      t2,0
        sw      a4,0(t4)
        li      a7,2
        li      a3,3
        li      a2,1
.L2:
        mulw    a5,a7,a7
        ble     a5,a3,.L3
        addiw   a7,a7,-1
        ble     t3,zero,.L23
        mv      a6,t5
        mv      a0,t4
        slli    t6,t3,2
        sh2add  t1,t3,t4
.L13:
        lw      a1,0(a6)
        blt     a7,a1,.L7
        lw      a5,0(a0)
        ble     a3,a5,.L9
.L8:
        addw    a5,a1,a5
        bgt     a3,a5,.L8
        sw      a5,0(a0)
.L9:
        beq     a3,a5,.L12
        addi    a0,a0,4
        addi    a6,a6,4
        bne     a0,t1,.L13
.L23:
        beq     t2,zero,.L6
        sw      t3,%lo(nSieve)(t0)
.L6:
#APP
# 19 "primes.c" 1
        rdtime s0
# 0 "" 2
#NO_APP
        lui     a1,%hi(.LC1)
        addi    a1,a1,%lo(.LC1)
        sub     s0,s0,s1
        li      a0,2
        call    __printf_chk
        fcvt.d.lu       fa5,s0
        lui     a1,%hi(.LC2)
        addi    a1,a1,%lo(.LC2)
        li      a0,2
        fmv.x.d a2,fa5
        call    __printf_chk
        ld      ra,24(sp)
        .cfi_remember_state
        .cfi_restore 1
        li      a0,0
        ld      s0,16(sp)
        .cfi_restore 8
        ld      s1,8(sp)
        .cfi_restore 9
        addi    sp,sp,32
        .cfi_def_cfa_offset 0
        jr      ra
.L3:
        .cfi_restore_state
        addiw   a7,a7,1
        j       .L2
.L7:
        li      a5,999
        bgt     t3,a5,.L11
        mulw    a5,a3,a3
        add     a4,t5,t6
        add     t6,t4,t6
        sw      a3,0(a4)
        addiw   t3,t3,1
        li      t2,1
        sw      a5,0(t6)
.L11:
        addiw   a2,a2,1
.L12:
        addiw   a3,a3,1
        j       .L2
        .cfi_endproc
.LFE55:
        .size   main, .-main
        .globl  nSieve
        .globl  sieve
        .globl  primes
        .bss
        .align  3
        .type   sieve, @object
        .size   sieve, 4000
sieve:
        .zero   4000
        .type   primes, @object
        .size   primes, 4000
primes:
        .zero   4000
        .section        .sbss,"aw",@nobits
        .align  2
        .type   nSieve, @object
        .size   nSieve, 4
nSieve:
        .zero   4
        .ident  "GCC: (Bianbu 13.2.0-23ubuntu4bb3) 13.2.0"
        .section        .note.GNU-stack,"",@progbits


Code: [Select]

# lscpu
Architecture:          riscv64
  Byte Order:          Little Endian
CPU(s):                8
  On-line CPU(s) list: 0-7
Model name:            Spacemit(R) X60
  Thread(s) per core:  1
  Core(s) per socket:  8
  Socket(s):           1
  CPU(s) scaling MHz:  100%
  CPU max MHz:         1600.0000
  CPU min MHz:         614.4000
Caches (sum of all):   
  L1d:                 256 KiB (8 instances)
  L1i:                 256 KiB (8 instances)
  L2:                  1 MiB (2 instances)


[SiliconWizard]

I ran the following benchmark, just to see: https://github.com/geerlingguy/top500-benchmark
Got  4.38 Gflops. In the list, the Sipeed Lichee Pi 3A obtained 4.95 Gflops. So, a bit lower performance here. Not sure where that comes from. That's about 13% difference, not as drastic as with your primes benchmark, which is more like 30%.

(Note: I put a fan over the heatsink for this benchmark - with just passive cooling, core temp was reaching 89°C.)

[SiliconWizard]

Here are my Geekbench 6 results: https://browser.geekbench.com/v6/cpu/14739771
From others with a K1, they seem to be in the top scores. So it doesn't look like this board has a particular problem, and the probability that the SoC is not a genuine K1 is very, very low. It's marked SpacemiT K1, it has the same dimensions, it talks like a K1, it's recognized as a K1...

There may be something slightly different compared to the LicheePi 3A - I suspect more a firmware difference than anything hardware. It's very likely that the FSBL is not the same version, and it's in there that the SoC gets initialized and the DDR controller trained.

The compiler seems to be the same (GCC 13.2.0), but it may have been built with different options? I'm not sure it should make any difference if the same revision, but just a thought. I'll make more "controlled" tests baremetal when I get to this point. I tried with Clang and already get better results with the primes microbenchmark than with GCC (tried with all -O options, doesn't make a big difference) - I got down to 26.8 billion cycles.

Anyway, next step is baremetal. The bootrom in the K1 supports fastboot, which makes it easier to test firmware - no need to flash it first, tests can be entirely in RAM.

[DiTBho]

I had a similar problem with the "neo" board (arm). The dram controller was set with more conservative values.

[SiliconWizard]

That's a possibility here, but the discrepancy with the primes microbenchmark can't be explained by DRAM access as it fits entirely into L1 cache, it's very small in code size and data size.
And, I also ran tinymembench which gave me results similar to what others have with the SpacemiT K1 @1.6 GHz.

Anyway, working on baremetal now. My custom-built FSBL (u-boot SPL) works and trains DDR RAM successfully @2400 MT/s. Now figuring out how to set up my test firmware.

[SiliconWizard]

Success with running baremetal on the K1.

I tested Bruce's primes baremetal and this is what I get: 23.8 billion clocks (14.870 s), CPI = 0.85. This is pretty close to what Bruce had gotten on the LicheePi 3A. Not sure what the culprit was on Bianbu when I tested it, but it may have been a compiler difference. I'm using mainline GCC 15.2 (cross compiling) for baremetal dev, the GCC compiler on Bianbu was 13.2 (same as Bruce, but he had apparently used an older version of Bianbu, mine was updated to 2.2.1). I have posted the disassembly, haven't compared yet with the disassembly I get with my baremetal environment. I just noticed that the compiler that ships with Bianbu doesn't look like mainline, it looks like a patched version that supports the 'spacemit-x60' CPU, which mainline GCC doesn't seem to support yet as of 15.2. So, the patches may explain the difference.

Anyway - that's looking pretty good. The initial plan was to use the following boot staging: BootROM -> FSBL -> OpenSBI -> Firmware, as I did on the CV1800B, but I did go for more direct without OpenSBI with the K1. The FSBL initializes the SoC and trains DRAM - that's pretty much all that's needed for "baremetal" stuff.

What I need to figure out now is 1/ how to use all cores (currently, my firmware runs only on the first core which is the one the FSBL runs on) and 2/ how to use the RCPU (extra RV32 core in the K1).

[SiliconWizard]

I figured out how to start secondary cores. On the K1, only hart 0 starts upon global reset (or maybe all start but the bootrom does park all secondary cores before jumping to the next stage, I don't know, we don't have the bootrom source code; but for power consumption reasons, that would make sense that the SoC would only start the first core (hart) to just run the bootrom code).

So cores 1..7 are in a deep sleep mode (that they call C2... the manual does not document power modes much, but that's what I figured out by reading the CORE_STATUS register). There is a series of registers to wake cores up from C1 or C2 (COREx_WAKEUP in the PMU), and another set of registers (undocumented, but found in the OpenSBI source code) to set the reset address of cores within a cluster (so cores 0..3 in cluster 0 share a single reset address, and ditto for cores 4..7 in cluster 1). When in C2 mode, waking up a core will power it back on and generate a reset, so it will start at the reset address defined in those registers.

So far, so good. That works. (Of course, if you use OpenSBI, it will do that for you. I decided to do without OpenSBI.)

I haven't tried with the RCPU core yet, but it's probably going to be relatively similar - there should be a register to get it out of reset and another to set its reset address, unless its reset address is 0, which is the start of the dedicated SRAM within its own address space. I'll see. Anyway, The RCPU is a Nuclei N308, and there's documentation for it from Nuclei (from what I can tell, better documentation than we have for the K1/X60).

Now, the thing that I'm still struggling with is cache coherency. I'm unable to get L1 cache coherency so far. Maybe there's a setting for that - a lot is left undocumented. Couldn't find it in OpenSBI source code so far though.
There is cache coherency at the L2 level, which I enabled (there is source code for that in OpenSBI, and it turns out that the K1 has a CCI-500 (or -550, unclear, but they are almost the same), which is an ARM IP, so documentation for it can be found from ARM, and the register layout is identical to that shown in OpenSBI's source code, so I have good reasons to think that they either licensed this ARM IP, or they cloned it somehow.

But between two cores within the same cluster? Unable to have cache coherency. I tried using fence instructions to no avail. The only thing that works so far is to manually flush / invalidate cache on either side, so that's really not good for sharing memory. The datasheet mentions "L1 cache supports MESI consistency protocol", so that should work - it doesn't seem to, so there must be something that isn't enabled. But no documentation, of course. So that'll be my next battle.

[SiliconWizard]

L1 cache coherency issue solved. The key is to set  some bits in the "ML2SETUP" CSR (0x7F0) (which is not documented so far): the four lower bits set cache snooping for each of the 4 cores of each cluster.
That could be seen in OpenSBI's source code in the spacemit_cold_boot_allowed() function. The fact it's set in this function was kind of confusing, but I guess the reason is that this function is guaranteed to be called for each detected core, even if adding some side-effects in it sounds weird given its role. Anyway, mainline OpenSBI has support for the K1 and does it too in the same function, but without comments. Fortunately, in the OpenSBI that's in the Bianbu repo, this is commented:

Code: [Select]

/* enable core snoop */
csr_set(CSR_ML2SETUP, 1 << (hartid % PLATFORM_MAX_CPUS_PER_CLUSTER));

So, with the corresponding bit set for each core in each cluster, cache coherency works as expected.

Apparently though, the CCI-500 unit in this SoC only has 2 slave interfaces (one per cluster), meaning that there is no cache coherency guaranteed for DMA, but that's not so uncommon.

Hopefully, SpacemiT will update the manual - as it is, it lacks quite a bit of information. Custom CSRs are undocumented, some peripherals/registers are not, or very briefly documented, and there are a few typos here and there (mainly with register addresses/offsets, which, to be fair, are easy to spot as they are mainly duplicate value that probably come from c/c). Fortunately, there's the available source code to figure out/reverse engineer what's missing in the manual.

[DiTBho]

there is no cache coherency guaranteed for DMA, but that's not so uncommon.

That's evil, as usual.
Already seen in MIPS4 systems.

[SiliconWizard]

Well, that's rather common, especially on "embedded" CPUs, which is how I'm going to use the K1. For "general-purpose" OSs, that's certainly a PITA. x86 platforms are mostly cache coherent when it comes to DMA (although there are some edge cases quirks), but many platforms are not.

Sure you can always use strictly non-cached areas for DMA, but that can be a huge performance penalty (unless you have access to some kind of TCM). That's not a route I personally want to take in general.

"Manual" management of cache for DMA buffers is not so bad, as long as there is no concurrent access between the CPU and DMA. Which is usually the case for DMA transfers themselves, but there is a caveat: DMA descriptors! The more advanced DMAs usually work with (chainable) descriptors in memory rather than simply just registers. And now, you *will* have concurrent access of the descriptors themselves, and with non-cache coherent memory, *that* is a royal pain.

The solution is either to put DMA descriptors into non-cached memory (which is not as bad performance-wise as putting the DMA data buffers themselves in non-cached memory), or put each descriptor into a single cache line, so align descriptors to a cache line and never share a cache line with several descriptors. The latter is the approach I've used on the K1 (so far working on Ethernet so, the DMA that works with the GMAC). Descriptors are 16-byte long, a cache line is 64-byte long, so by putting one descriptor per cache line, I lose a bit of memory (but that's pretty negligible here), but that makes things reliable and much easier to handle. Using contiguous descriptors was awful to debug. Now that's possible because the GMAC DMA can be configured with a non-zero increment between consecutive descriptors. Haven't looked yet if the general DMA allows the same thing, hopefully so.

[DiTBho]

Well, that's rather common, especially on "embedded" CPUs, which is how I'm going to use the K1. For "general-purpose" OSs, that's certainly a PITA.

"MIPS4"(1) were(2) not "embedded systems" but rather "workstations" and "servers".
Developing the Linux kernel on these systems is very time-consuming, so much so that only experts are willing to take on source development.

This is why Linux is not suitable for SGI/Mips and is very limited. Furthermore, there is no documentation. It's all experimental.


(1) CPU[]={ R10000, R12000, R14000, R16000 }
(2) in the 90s and early 2000s

[SiliconWizard]

Well, for the SpacemiT K1, they have done it - the Linux port seems solid. But it's probably not the most efficient.

I haven't looked at much of the Linux kernel for it yet, apart from specific drivers (such as emac) to help write my own, but DMA is handled via kernel calls so that's hidden there, and I haven't looked (yet) at the part where they handle the DMA. Maybe they allocate non-cached memory, not sure. U-boot has its own emac driver, which is relatively similar (likely written by the same devs) but closer to the metal, and they do flush and invalidate DMA buffers as needed. For DMA descriptors, instead of placing one descriptor per cache line as I did, they seem to be rewriting all descriptors lying in a whole cache line every time they change one, which looks pretty inefficient. I prefer wasting a few tens of bytes and not having to deal with neighboring descriptors every time I modify one.

For most simple peripherals (GPIO, UART, SPI, I2C, Timers, ...) documentation is enough to get it done, but the 3 USB controllers are essentially undocumented. From what I gathered so far, the USB 3 controller is a DWC3 IP, so that's where to look. For the 2 USB 2 controllers, I don't know yet. Maybe DWC2, maybe something else. I'll see. The GPU is undocumented, but I'm sure this is under NDA with Imagination (the IP provider), and the latter provides a SDK, but the main library is provided in binary form for Linux: https://developer.imaginationtech.com/downloads/ . Not sure I'll be able to do much with it in a baremetal way. The PCIe controllers are also undocumented, but source code for drivers should be in the Linux port.

Note that SpacemiT documents its Linux port, which makes it easier to find the corresponding source code, see  here: https://bianbu-linux.spacemit.com/en/development_guide/peripheral_driver/