Verifying RISC-V compliance

[laugensalm]

Hi there,

I'm hoping someone with insights can give me a hint on that one:
I'd like to eventually be able to verify a proprietary RISC-V (rv32i/c compatible) design towards certain compliance, with the option of demonstrating correct behaviour under plenty of possible constructs and insn sequences:

My research so far yielded this:
- github.com/riscv/riscv-compliance and ../riscv-tests
- github.com/google/riscv-dv

Looks like some of them depend on specific Verilog based sim environments - unfortunately, I'm kinda bound to the VHDL side (GHDL simulator, in particular).
Any idea about:
- Which of the above is the most 'state of the art' to work with?
- Which might be the easiest to extend for any kind of simulator?

I have an interface to download code to (virtual) RAM and read back values in the running simulation, for example. However, as this could turn complex, I'd like to take it on the automated way to hardware as well (where JTAG ICE would get into place).

Any other opinions on the best way to go are welcome.

[SiliconWizard]

Do you have access to the RISC-V core's code?

[laugensalm]

Sure, i got VHDL which I'm sticking into a virtual SoC, i.e. I have the 'virtual machine'. Just wondering what's the best approach to make one compliance checker of the above speak to it. I used to access the simulation over a pipe or network from qemu, for example.

[SiliconWizard]

Unfortunately, most RISC-V open projects are written in Verilog, SV or more exotic HDLs; very few in VHDL. And the ones I've found in VHDL didn't seem to include any real test code to speak of.

Indeed, any Verilog-based test code will be unusable with GHDL.

One of the regulars here works at SiFive, so he may be able to help you.

[laugensalm]

The github/riscv/riscv-tests are just pure macro-featured assembly, looks like they would work with my unit testing setup. I've done a similar dance before (this RISC-V design used to be a MIPS). So what I'm targetting at is just making one of these regress-test suites download the code into the VM or even the synthesized HW and running the tests on it (with some remote control involved, obviously). I believe this should - in principle - be HDL agnostic, but hopefully some of the experts can shed more light on that.

[brucehoult]

I'm not sure if it's what you're asking but we have a special memory-mapped peripheral device called TestFinisher with a single 4 byte register that CPU core tests use to write their result to and terminate the test. Sadly, I can't show you TestFinisher.sv that is used to add it to the hardware (whether FPGA or SoC), but here is the implementation in qemu:

https://github.com/qemu/qemu/blob/master/hw/riscv/sifive_test.c

[laugensalm]

Thanks for the link, I'm actually using in circuit emulation (via a stripped down CP0/CSR) to pass messages between core and external test logic.
So once I have a binary test snippet, I can run it on the target and verify results for some rather simple tests like hazard scenarios. Would you consider passing the riscv-tests suite 'sufficient' for compliance? I'm not looking for safety, but I want to go a bit further eventually than 'program just working'. On one hand it's like "not breaking stuff", when introducing funky features, eventually I'd like the CI setup to tell me. For that the riscv-tests suite seems totally sufficient, but I was wondering about more complex coverage.

[brucehoult]

More complete and rigorous compliance testing is a work in progress that a committee is working on. Meantime, people are seeming to get by with riscv-tests.

As you say, it seems to be sufficient in practice for "that change didn't seem to break anything". If people start to try to get a RISC-V logo on things that are maliciously and subtly broken then that will be something else entirely.

[laugensalm]

Thanks, Bruce. Would that be the authoring team of this repo: https://github.com/riscv/riscv-compliance ?

I've got to spool through the riscv/riscv-tests with minor changes of some macros (due to my stripped down setup). Looks like a PASS. However there are some tricky scenarios where I had to actually code a lot of test routines, for example when you're going into ICE (single stepping, etc.) while having IRQs enabled or when any kind of exception occurs. So the hard part will actually be when a 'compliant' CSR is implemented and the entire event behaviour is to be verified - and this is where I had to cheat for now. I guess the CSR 'compliance' is still left open?
Also, the more extensive testing I'm interested in is handling/coverage of unsupported insns or just garbage. I used to shoot pseudorandom sequences at another arch having only 8 bit opcodes (i.e.not taking months to complete). I wonder what would serve 'good enough' for 32 bit RISC-V opcodes.

[brucehoult]

Thanks, Bruce. Would that be the authoring team of this repo: https://github.com/riscv/riscv-compliance ?

Yes.

I've got to spool through the riscv/riscv-tests with minor changes of some macros (due to my stripped down setup). Looks like a PASS. However there are some tricky scenarios where I had to actually code a lot of test routines, for example when you're going into ICE (single stepping, etc.) while having IRQs enabled or when any kind of exception occurs. So the hard part will actually be when a 'compliant' CSR is implemented and the entire event behaviour is to be verified - and this is where I had to cheat for now. I guess the CSR 'compliance' is still left open?

I don't know what the situation is with CSRs. I suggest you raise an issue about that in the github repo, if you are able to.

Also, the more extensive testing I'm interested in is handling/coverage of unsupported insns or just garbage. I used to shoot pseudorandom sequences at another arch having only 8 bit opcodes (i.e.not taking months to complete). I wonder what would serve 'good enough' for 32 bit RISC-V opcodes.

As RISC-V is intended to be extensible with custom instructions, I believe the plan is that official RISC-V compliance testing will only test that instructions in the base ISA and standard extensions that are claimed to be supported work correctly. I believe it will *not* test unsupported or non-standard instructions or garbage.

Of course a manufacturer of a CPU will want to do more extensive testing.

[laugensalm]

Hi Bruce,

got your point about insn extending, makes sense.
I've been diving more into the CSR details and realized it would make my setup more complicated plus use more resources. A few implementations I looked at don't do things the standard CSR way, so I ended up likewise, just emulating the minimum to be able to remote control the core in some way.

For those interested, I added that setup into my build environment (https://github.com/hackfin/MaSoCist/). A 'make all test' (after doing the docker container dance) will build the (VHDL) simulation executable and a Python script is downloading/running the modified riscv-tests into it through a virtual ICE debug logic.