RP2040 Pi Pico: buggy J-Link/Ozone, yet OpenOCD works

[mblinov]

Hi all, I'm facing a really frustrating issue: I am trying to run under debug one of the pico-examples projects, specifically the hello_uart.c example.

Now, I've got two debug probes in front of me:

• A Pi Pico that is flashed with the latest picoprobe binary, using the latest freshly-built openocd that is specific for the picoprobe
• A J-Link Edu mini

The Picoprobe works without a hitch: The J-Link does not. The J-Link can load the binary, single step for a bit, but it very quickly throws a HardFault exception, which I'm having real trouble debugging.

The point at which the HardFault is thrown appears to not be consistent.

Judging from the internet (and SEGGER's own website) this should work without a hitch, so I'm fairly confident I'm doing something wrong - the only trouble is, as far as I can tell, I'm doing everything by the book - would love if someone more experienced with debug probes would suggest something.


The (working) Picoprobe OpenOCD flow

Now, as the title says, the OpenOCD -> picoprobe flow works perfectly. The flow works like this:

1). Open terminal A (MSYS2), and invoke

Code: [Select]

$ openocd-install/bin/openocd.exe \
    -f openocd-install/share/openocd/scripts/interface/cmsis-dap.cfg \
    -f openocd-install/share/openocd/scripts/interface/picoprobe.cfg -f openocd-install/share/openocd/scripts/target/rp2040.cfg

The server then duly responds:

Code: [Select]

Open On-Chip Debugger 0.11.0-g8e3c38f78 (2023-02-11-21:45)
Licensed under GNU GPL v2
For bug reports, read
        http://openocd.org/doc/doxygen/bugs.html
Warn : Interface already configured, ignoring
adapter speed: 5000 kHz

Info : auto-selecting first available session transport "swd". To override use 'transport select <transport>'.
Info : Hardware thread awareness created
Info : Hardware thread awareness created
Info : RP2040 Flash Bank Command
Info : Listening on port 6666 for tcl connections
Info : Listening on port 4444 for telnet connections
Info : Using CMSIS-DAPv2 interface with VID:PID=0x2e8a:0x000c, serial=E66118604B1F5E21
Info : CMSIS-DAP: SWD  Supported
Info : CMSIS-DAP: FW Version = 2.0.0
Info : CMSIS-DAP: Interface Initialised (SWD)
Info : SWCLK/TCK = 0 SWDIO/TMS = 0 TDI = 0 TDO = 0 nTRST = 0 nRESET = 0
Info : CMSIS-DAP: Interface ready
Info : clock speed 5000 kHz
Info : SWD DPIDR 0x0bc12477
Info : SWD DLPIDR 0x00000001
Info : SWD DPIDR 0x0bc12477
Info : SWD DLPIDR 0x10000001
Info : rp2040.core0: hardware has 4 breakpoints, 2 watchpoints
Info : rp2040.core1: hardware has 4 breakpoints, 2 watchpoints
Info : starting gdb server for rp2040.core0 on 3333
Info : Listening on port 3333 for gdb connections

2). Open terminal B (cmd.exe), and open GDB:

Code: [Select]

C:\Users\Maxim\prj\pi-pico-blah>arm-none-eabi-gdb.exe pico-examples-build-debug\uart\hello_uart\hello_uart.elf

Connect to the OpenOCD GDB server interface on port 3333:

Code: [Select]

(gdb) tar extended-remote :3333

Load the binary:

Code: [Select]

(gdb) load
Loading section .boot2, size 0x100 lma 0x10000000
Loading section .text, size 0x4200 lma 0x10000100
Loading section .rodata, size 0xfa0 lma 0x10004300
Loading section .binary_info, size 0x20 lma 0x100052a0
Loading section .data, size 0x22c lma 0x100052c0
Start address 0x100001e8, load size 21740
Transfer rate: 4 KB/sec, 3623 bytes/write.
(gdb)

Now, I can run the program to completion

Code: [Select]

(gdb) c
Continuing.
target halted due to debug-request, current mode: Thread
xPSR: 0x01000000 pc: 0x00000138 msp: 0x20041f00

Thread 1 received signal SIGTRAP, Trace/breakpoint trap.
_exit (status=0)
    at C:/Users/Maxim/prj/pi-pico-blah/pico-sdk/src/rp2_common/pico_runtime/runtime.c:186
186             __breakpoint();
(gdb)

(It is expected for us to end up in _exit, since the main() function in this case returns after printing some text.)


The (non-working) J-Link flow

The J-Link tools however are constantly throwing HardFaults all over the place. I've tried using two tools:

• J-Link GDB server
• Ozone

I'm using the latest J-Link as of writing (version 7.84f DLLs, and version 3.28c for Ozone.)

Here is what a typical J-Link GDB server session looks like:

First, setup the server:


After launching:


This time ofcourse I don't have to open a separate OpenOCD terminal. In cmd.exe, I open the GDB client exactly the same way, exactly the same binary:

Code: [Select]

C:\Users\Maxim\prj\pi-pico-blah>arm-none-eabi-gdb.exe pico-examples-build-debug\uart\hello_uart\hello_uart.elf

And attach to the port as specified by the J-Link GDB server (port 2331):

Code: [Select]

(gdb) tar extended-remote :2331
Remote debugging using :2331
__breakpoint ()
    at C:/Users/Maxim/prj/pi-pico-blah/pico-sdk/src/rp2_common/pico_platform/include/pico/platform.h:269
269         __asm__("bkpt #0");
(gdb)

Ok, fine. I should specify at this point: I have not powered off the Pi Pico under debug: I've literally hot-swapped the SWD connectors from the Picoprobe over to the J-Link. Nothing has been power cycled. (Actually, reading this back: it *is* weird that it reports a different source line...)

So now I load the binary:

Code: [Select]

(gdb) load
Loading section .boot2, size 0x100 lma 0x10000000
Loading section .text, size 0x4200 lma 0x10000100
Loading section .rodata, size 0xfa0 lma 0x10004300
Loading section .binary_info, size 0x20 lma 0x100052a0
Loading section .data, size 0x22c lma 0x100052c0
Start address 0x100001e8, load size 21740
Transfer rate: 732 KB/sec, 3623 bytes/write.

All the numbers exactly the same, fine. Now I just continue:

Code: [Select]

(gdb) c
Continuing.

Program received signal SIGTRAP, Trace/breakpoint trap.
0xfffffffe in ?? ()

And there it is. We don't end up in _exit as last time, but in a reset vector of some kind? (showing my ignorance here.)

Listing the backtrace gives

Code: [Select]

(gdb) bt
#0  0xfffffffe in ?? ()
#1  <signal handler called>
#2  0x60000200 in ?? ()
#3  0x10000264 in hold_non_core0_in_bootrom ()
    at C:/Users/Maxim/prj/pi-pico-blah/pico-sdk/src/rp2_common/pico_standard_link\crt0.S:322
Backtrace stopped: previous frame identical to this frame (corrupt stack?)


My own observations

Really interesting aside: The entry point symbol is called _entry_point. If I put a breakpoint at main, and print the program counter with (gdb) p /x $sp, the value is different depending on whether the GDB server is OpenOCD or J-Link:

For the OpenOCD, upon entry to main the stack pointer was 0x20042000:

Code: [Select]

(gdb) load
Loading section .boot2, size 0x100 lma 0x10000000
Loading section .text, size 0x4200 lma 0x10000100
Loading section .rodata, size 0xfa0 lma 0x10004300
Loading section .binary_info, size 0x20 lma 0x100052a0
Loading section .data, size 0x22c lma 0x100052c0
Start address 0x100001e8, load size 21740
Transfer rate: 4 KB/sec, 3623 bytes/write.
(gdb) i b
No breakpoints or watchpoints.
(gdb) b main
Breakpoint 1 at 0x10000364: file C:/Users/Maxim/prj/pi-pico-blah/pico-examples/uart/hello_uart/hello_uart.c, line 22.
Note: automatically using hardware breakpoints for read-only addresses.
(gdb) c
Continuing.
target halted due to debug-request, current mode: Thread
xPSR: 0x01000000 pc: 0x00000138 msp: 0x20041f00

Thread 1 hit Breakpoint 1, main ()
    at C:/Users/Maxim/prj/pi-pico-blah/pico-examples/uart/hello_uart/hello_uart.c:22
22      int main() {
(gdb) p /x $sp
$1 = 0x20042000

And for the J-Link, upon entry to main the stack pointer was 0x20041ff0:

Code: [Select]

(gdb) load
Loading section .boot2, size 0x100 lma 0x10000000
Loading section .text, size 0x4200 lma 0x10000100
Loading section .rodata, size 0xfa0 lma 0x10004300
Loading section .binary_info, size 0x20 lma 0x100052a0
Loading section .data, size 0x22c lma 0x100052c0
Start address 0x100001e8, load size 21740
Transfer rate: 758 KB/sec, 3623 bytes/write.
(gdb) c
Continuing.

Thread 2 hit Breakpoint 1, main ()
    at C:/Users/Maxim/prj/pi-pico-blah/pico-examples/uart/hello_uart/hello_uart.c:22
22      int main() {
(gdb) p /x $sp
$3 = 0x20041ff0

(That is, assuming that I can get to main before it throws an exception.)

[mblinov]

P.S. if someone could *please* tell me what the difference between RP2040_M0_0 and RP2040_M0_1 in the J-Link MCU selector is, I will be forever indebted:

[Foxxz]

Is that maybe which CPU core you want to debug?

[mwb1100]

P.S. if someone could *please* tell me what the difference between RP2040_M0_0 and RP2040_M0_1 in the J-Link MCU selector is, I will be forever indebted:

Just a guess, but each entry might be for one of the 2 cores on the RP2040

[mwb1100]

Have you read Segger's wiki article on the RP2040 - https://wiki.segger.com/Raspberry_Pi_RP2040

When working with the RP2040, the following must be considered in regard to its ROM bootloader:
The ROM BTL expects a "flash second stage" (aka. "2nd stage bootloader") in the external QSPI flash.
The ROM BTL verifies that such a valid 2nd stage BTL is present by calculating a CRC over the first 252 bytes of the QSPI flash and comparing it against the 4 bytes following those 252 bytes. Only if the calculated CRC matches the CRC value in flash, the ROM BTL will jump to user application code (i.e. the 2nd stage BTL).

The 2nd stage BTL is not part of the application image and as such can be adapted by the user. It usually performs some additional setup to enable XIP from the QSPI flash or similar before jumping to the "actual" start of application.

Maybe the OpenOCD .cfg files are taking care of this somehow and it's not being done for the J-Link?

One of the things that the second stage bootloader does is set up the execute in place (XIP) environment for the flash that's on the board.  I imagine that if that's not set up correctly, the kind of crash you're seeing might be the result.

Also, I see that in the OpenOCD example when it gets to the breakpoint at main, it's on "Thread 1".  But for the J-Link it's on "Thread 2".  Don't know what that means, but it seems fishy.

Of course, I've only spent about 10 minutes reading the Segger article an some of the PR2040 datasheet.  So I could be spouting non-sense.   But if you don't know how the 2nd stage BTL is working, it might be a place to start looking.

[mino-fm]

One of the things that the second stage bootloader does is set up the execute in place (XIP) environment for the flash that's on the board.  I imagine that if that's not set up correctly, the kind of crash you're seeing might be the result.

As far as I can remember 2. stage bootloader is included in Segger's examples.
I tried Segger's IDE and Ozone but put it away very soon. I don't know anything about OpenOCD.

@mblinov
Did you connect /reset-pin of J-Link to run-pin of RP2040?
Do you like to test another IDE I'm working with? It is EWARM from IAR. I'm using J-Link edu-mini too and could give you some demo examples. Debugging works very fine.

Sometimes debugging is critical because of RP2040's bootloader, which is always running before reaching main(). It seems to me, programm is starting without stop at main() so hard faults could happen before the degugger could tell you anything.
Without PowerOn there is no reset of GPIO, UART, ... so you have to do it by your own. Sometimes this can be very nasty.

[bson]

All the numbers exactly the same, fine. Now I just continue:

Code: [Select]

(gdb) c
Continuing.

Program received signal SIGTRAP, Trace/breakpoint trap.
0xfffffffe in ?? ()


It's in _exit.  The problem is the gdb server in this case doesn't properly cross exceptions on the call stack.  If you do a stack backtrace ('bt') in gdb one will show the exception boundary, the other will just display nonsense.

It's also be plausible that even if their own debugger uses the gdb debug server, it might not rely on it to walk the stack frame chain so they don't really use this server functionality themselves and don't see a problem.  Then when a real gdb doesn't work right they assume it's just gdb.

[enz]

Did you ever solve the problem?

I am asking because i ran into the exact same issue today and found the solution in the Segger Wiki after some digging around.