[RP2354] [PI-PICO2] Custom RP2354 board: code never reaches main(), PC stuck ins

[Abidarian]

Hi all,

First custom board I've designed and I'm hitting a brick wall on bring-up. Hoping someone here with custom RP2350-family bring-up experience can spot what I've missed. I'd really appreciate a sanity check on the schematic too if anyone has the time.

## The board and tooling

It's a custom RP2354 board. SWD, QSPI flash, RUN, and USB_BOOT (QSPI SS) are all broken out. For debug I'm using a Pi Pico 2 W flashed with debugprobe firmware, talking to OpenOCD on a Raspberry Pi host (cmsis-dap.cfg + target/rp2354.cfg). Firmware is built with the pico-sdk, `-DPICO_BOARD=pico2`.

Schematic is attached. Apologies in advance for the GPIO net names on the breakout, they're misleading and already fixed in the next rev.

## What's happening

I flash a trivial blinky-style program over SWD. After reset, nothing happens on the pin I expect to go high (multimeter reads ~0.20 V on it). When I halt the core and read the PC, it's sitting deep inside `xosc_init` from the SDK's runtime init, well before `main()` ever runs. A small RAM log buffer I write to from the very first line of `main()` is completely empty after halt, which confirms `main()` is not being reached.

## Minimal test program

```c
#include "pico/stdlib.h"
#include "hardware/gpio.h"

int main(void) {
    gpio_init(19);
    gpio_set_dir(19, GPIO_OUT);
    gpio_put(19, 1);

    gpio_init(20);
    gpio_set_dir(20, GPIO_OUT);
    gpio_put(20, 0);

    while (true) { tight_loop_contents(); }
}
```

CMakeLists:

```cmake
cmake_minimum_required(VERSION 3.13)
include($ENV{PICO_SDK_PATH}/external/pico_sdk_import.cmake)
project(charlie_demo C CXX ASM)
set(CMAKE_C_STANDARD 11)
set(CMAKE_CXX_STANDARD 17)
pico_sdk_init()
add_executable(charlie_demo main.c)
target_link_libraries(charlie_demo pico_stdlib hardware_gpio)
pico_add_extra_outputs(charlie_demo)
pico_enable_stdio_usb(charlie_demo 0)
pico_enable_stdio_uart(charlie_demo 0)
```

Built clean, no warnings:

```bash
cd ~/charlie_demo
rm -rf build && mkdir build && cd build
cmake .. -DPICO_BOARD=pico2
make -j4
```

## OpenOCD halt and register dump

```text
$ openocd -f interface/cmsis-dap.cfg \
          -f target/rp2354.cfg \
          -c "adapter speed 5000" \
          -c "init" \
          -c "halt" \
          -c "dump_image /tmp/ramlog.bin 0x20000000 1024" \
          -c "mdw 0xd0000010" \
          -c "mdw 0xd0000030" \
          -c "reg pc" \
          -c "exit"

Warn : [rp2354.cm0] target was in unknown state when halt was requested
Info : SWD DPIDR 0x4c013477
Warn : [rp2354.cm1] target was in unknown state when halt was requested
Info : [rp2354.cm0] external reset detected
dumped 1024 bytes in 0.004106s (243.546 KiB/s)
0xd0000010: 00000000
0xd0000030: 00000000
pc (/32): 0x10000f66
```

PC at `0x10000f66` is inside the XOSC startup busy-wait. The strange part: when I halt, the `STABLE` bit in `XOSC.STATUS` is actually set, but the core is still spinning in that loop.

## What I've tried so far

1. Cleared the `ISO` bit in `PADS_BANK0` for the GPIOs (pad isolation defaults on, has to be cleared after pad config). This nudged the PC past the previous stuck address, so the suggestion was definitely on the right track.
2. Bumped `XOSC_STARTUP_DELAY_MULTIPLIER` to 128. No change.
3. Verified `IO_BANK0: GPIOx_STATUS` and confirmed I'm not accidentally overriding the output via `GPIOx_CTRL`.

After step 1, the PC moved but it's still in `xosc_init`, just at a slightly different address:

```text
pc (/32): 0x10000d42

0x40048000 (XOSC_BASE):     00fabaa0 81001000 77616b65 00001780 00000000
0x40050000 (PLL_SYS_BASE):  00000001 0000002d 00000000 00077000
0xd0000010 (SIO GPIO_OUT):  00000000
0xd0000030 (SIO GPIO_OE):   00000000
```

Reading those: `XOSC.CTRL = 0x00fabaa0` and `XOSC.STATUS = 0x81001000` (MSB high, so STABLE is set). SIO `GPIO_OUT` and `GPIO_OE` are both still zero, so we are nowhere near the GPIO config code. PLL_SYS looks like it hasn't been kicked either.

## Where I'm stuck

Tomorrow I'll get a scope on XIN to see whether the crystal is actually oscillating. But before I do that, I'd like to ask the assembled wisdom:

1. Has anyone seen this exact failure mode on a custom RP2350-family board, where `XOSC.STATUS` reads STABLE but the SDK still hangs in `xosc_init`? Is there something I'm misreading about that STATUS register?
2. What are the usual culprits on a custom board that produce a hang this early? Things I'm specifically wondering about: load capacitor mis-sizing on XIN/XOUT, supply sequencing or insufficient decoupling on `DVDD` / `VREG_VOUT`, OTP being in an unexpected state from the factory, or the QSPI flash boot not completing cleanly and somehow leaving the chip in a half-initialised state.
3. Any tips on how to distinguish "crystal genuinely not oscillating" from "crystal is fine, something upstream (resets, PSM, regulator) is keeping us stuck" purely from SWD register reads, so I can narrow it down before reaching for the scope?

Happy to post additional register dumps, the linker map, BOM, or the layout if useful. Schematic attached below.

Thanks in advance, any sanity check would be appreciated.

[brucehoult]

Almost certainly the oscillator, not any software thing. Those chips are incredibly fussy about the crystal and especially inductor, and even freaking orientation of the inductor.

[Abidarian]

I'll be pulling up an oscillator on this thing to check if it's even generating any clock cylces or not. If things are not right, maybe try switching the capacitors or the crystal itself. What are my other options?

[aeg]

If the inductor turns out to be the problem, people have had success running these from external LDOs instead of the internal buck regulator.

[phil from seattle]

Verify that you have 1.1V and 3.3V (or 1.8 if doing that) on the appropriate pins.  The 2350 isn't terribly fussy about it's xtal but it wouldn't hurt to go with the abracon spec'd one and the exact load cap values.  If you have an o'scope, take a look at the osc pins.  You should see a clean signal at 12 mhz.

Also, can you hook up USB?  Probably not because your schematic doesn't appear to have any traces brought out.  In general, it's good idea to do that, even if just to a header.  If nothing else, it makes loading firmware easier when it's not hooked up to a debugger.

[Abidarian]

Verify that you have 1.1V and 3.3V (or 1.8 if doing that) on the appropriate pins.  The 2350 isn't terribly fussy about it's xtal but it wouldn't hurt to go with the abracon spec'd one and the exact load cap values.  If you have an o'scope, take a look at the osc pins.  You should see a clean signal at 12 mhz.

Also, can you hook up USB?  Probably not because your schematic doesn't appear to have any traces brought out.  In general, it's good idea to do that, even if just to a header.  If nothing else, it makes loading firmware easier when it's not hooked up to a debugger.

I checked the voltage on all the IOVDD AND DVDD pins thei have their respective 3.3V and 1.1V potentials. Also the inductor I am using doesn't have any specific direction mentioned (there are no marking on the package).

I don't have USB pads brought out but I am trying to debug and flash using the SWD.

I am yet to read the OSC pins with an oscillator, I'll hopefully do it today. Also I noticed the follwoing this today -
When I try to put the MCU in bootsel using the pads exposed nothing really happens,
When I try to bootsel the MCU stays at the initial state only
pc (/32): 0x00006086
0x40048000: 00fab000 81001000 77616b65 000000c3 00000000
0x40050000: 00000001 0000002d 00000000 00077000
0x40058000: 80000001 00000000 00000064 00055000
0xd0000010: 00000000
0xd0000030: 00000000
0xd0000004: 00000000
The PC is in BootROM

when I try to flash uf2 using SWD, the sdk writes configs to the MCU but then it gets stuck at XOSC_INIT function
pc (/32): 0x10000d42
0x40048000: 00fabaa0 81001000 77616b65 0000011a 00000000
0x40050000: 00000001 0000002d 00000000 00077000
0x40058000: 00000001 0000002d 00000000 00077000
0xd0000010: 00000000
0xd0000030: 00000000
0xd0000004: 00000000

Also I tried running the GPIOs on the PCB without the crystal by writing to the registers, and I could drive the required GPIOs HIGH or LOW.

[Dan N]

Have you read the RP2350 hardware design guide?  It gives very detailed info on all of the potential issues others posted above.

[Abidarian]

I revisited this manual and found out that the inductor part number on my PCB is different for the one mentioned in the manual. Both are 3.3uH but part numbers are different. I am using MLZ2012M3R3ATD69 and the one in the manual is AOTA-B201610S3R3-101-T. Could this be causing the problem? I am using the 15pF capacitors and the tracks are a couple of mm distant from the MCU, could this add up significant parasitic capacitance? I am thinking if I should try replacing it with lower capacitance values?

[squadchannel]

any board layout image?

[Abidarian]

any board layout image?

Yes I am attaching the images with this post. I am sorry if this isn't what you were asking for.
Here is the 3D view of the PCB bottom where all the components are

Here is the bottom layer Cu

[squadchannel]

 

[Abidarian]

I am sorry for disappointing, new to this stuff. Let me know how can I not annoy people who are helping me here

[phil from seattle]

I revisited this manual and found out that the inductor part number on my PCB is different for the one mentioned in the manual. Both are 3.3uH but part numbers are different. I am using MLZ2012M3R3ATD69 and the one in the manual is AOTA-B201610S3R3-101-T. Could this be causing the problem? I am using the 15pF capacitors and the tracks are a couple of mm distant from the MCU, could this add up significant parasitic capacitance? I am thinking if I should try replacing it with lower capacitance values?

You are doing fine.  If anyone is annoyed, they can ignore your posts.

If you are getting 1.1V, the inductor is doing it's job.  There was a lot controversy over them putting a buck regulator on the chip and having an inductor so close to the chip.  RPI recommends using the Abracon one and orienting with the "correct" polarity (dot pin to VREG_LX, iirc). Incorrect placement can lead to instability according to the RPI engineers. I use an LDO instead - turns out to be cheaper than the Abracon inductor and basically no noise.  Take a look at 1.1V on a scope to see if it is reasonable.  But, I suspect this is not your problem.

Yes, I understood your dev/debug setup but was just saying that in the future, bringing out the USB traces is cheap and gives you more options.

Again, look at XIN on an O'scope.  Your layout of the Osc section looks straight forward.

Now, I noticed some things about your layout that might be an issue.  In particular, you have traces under the RP235x and no apparent ground connection for the RP235x. See my layout for the 2350B below.  2 layer board, continuous ground plane on the bottom. Note how I have a center ground with 9 vias to the bottom ground plane. You don't and I wonder how the ground slug (aka Pin 65) is connected to ground. Then, I have a ring of Dvdd and an outer ring of IOvdd.  No signal traces under the chip.

Also, I see a lot of vias that appear not to be connected to your ground plane. What's up with that? Do you have disconnected ground planes on either side of the board? You really want to have every ground pour tightly connected to every other one.

[Abidarian]

I revisited this manual and found out that the inductor part number on my PCB is different for the one mentioned in the manual. Both are 3.3uH but part numbers are different. I am using MLZ2012M3R3ATD69 and the one in the manual is AOTA-B201610S3R3-101-T. Could this be causing the problem? I am using the 15pF capacitors and the tracks are a couple of mm distant from the MCU, could this add up significant parasitic capacitance? I am thinking if I should try replacing it with lower capacitance values?

You are doing fine.  If anyone is annoyed, they can ignore your posts.

If you are getting 1.1V, the inductor is doing it's job.  There was a lot controversy over them putting a buck regulator on the chip and having an inductor so close to the chip.  RPI recommends using the Abracon one and orienting with the "correct" polarity (dot pin to VREG_LX, iirc). Incorrect placement can lead to instability according to the RPI engineers. I use an LDO instead - turns out to be cheaper than the Abracon inductor and basically no noise.  Take a look at 1.1V on a scope to see if it is reasonable.  But, I suspect this is not your problem.

Yes, I understood your dev/debug setup but was just saying that in the future, bringing out the USB traces is cheap and gives you more options.

Again, look at XIN on an O'scope.  Your layout of the Osc section looks straight forward.

Now, I noticed some things about your layout that might be an issue.  In particular, you have traces under the RP235x and no apparent ground connection for the RP235x. See my layout for the 2350B below.  2 layer board, continuous ground plane on the bottom. Note how I have a center ground with 9 vias to the bottom ground plane. You don't and I wonder how the ground slug (aka Pin 65) is connected to ground. Then, I have a ring of Dvdd and an outer ring of IOvdd.  No signal traces under the chip.

Also, I see a lot of vias that appear not to be connected to your ground plane. What's up with that? Do you have disconnected ground planes on either side of the board? You really want to have every ground pour tightly connected to every other one.

Yes I will add the USB atleast for the while I am still developing this board. (I hadn't anticipated the need for this earlier)

I'll have access to the O'scope later in the day, I'll share the results once I have my hands on it.

Yes I forgot to expose the pad for bottom ground slug, the vias all over the board are from the charliplexed LED matrix from the front face. this is a 4 layer board and the Ground planes are on the third and the fourth layer. I snuck those vias under the MCU in an attempt to try and manage the matrix signal pin to the respective GPIO. I initially was trying to build the exact same dvdd and iovdd but couldn't fit in the same while managing the viases.

[phil from seattle]

Yes I forgot to expose the pad for bottom ground slug, the vias all over the board are from the charliplexed LED matrix from the front face. this is a 4 layer board and the Ground planes are on the third and the fourth layer. I snuck those vias under the MCU in an attempt to try and manage the matrix signal pin to the respective GPIO. I initially was trying to build the exact same dvdd and iovdd but couldn't fit in the same while managing the viases.

Since the slug is the only ground connection for the IC, I think that is your problem. 

If you redo the board, I highly recommend you use a similar scheme for ground as I did and put no signals under the IC. You really should spend time with The Hardware Design guide. Also take a look at the minimal design project.  I used those resources when designing my board and it worked first time.

[Abidarian]

Yes I forgot to expose the pad for bottom ground slug, the vias all over the board are from the charliplexed LED matrix from the front face. this is a 4 layer board and the Ground planes are on the third and the fourth layer. I snuck those vias under the MCU in an attempt to try and manage the matrix signal pin to the respective GPIO. I initially was trying to build the exact same dvdd and iovdd but couldn't fit in the same while managing the viases.

Since the slug is the only ground connection for the IC, I think that is your problem. 

If you redo the board, I highly recommend you use a similar scheme for ground as I did and put no signals under the IC. You really should spend time with The Hardware Design guide. Also take a look at the minimal design project.  I used those resources when designing my board and it worked first time.

I too see it now, I ignored the pad earlier thinking it is just a thermal pad and wouldn't be causing much trouble. (Although I had planned to connect it to the GND, it slipped my mind.)
replicating the exact minimal layout would be a difficult and tricky job. I'll try doing it either ways. The problem are the vias from the charliplexed matrix being all over the board.

Also I have ordered the suggested Inductors, I'll try both, replacing the current one and switching to an LDO. In the meanwhile I am thinking of hacking a bodge wire to the underpad GND and connecting it to the board GND. Testing all these changes, I'll order a new board.

any more things I should take care of?

[aeg]

Your schematic shows a connection to the ground pad. Did you run a netlist comparison on your layout? (Might be part of DRC depending on your software.)

[Abidarian]

Your schematic shows a connection to the ground pad. Did you run a netlist comparison on your layout? (Might be part of DRC depending on your software.)

I did run a DRC and didn't find any netlist error. This might be due to the fact that I edited the footprint to remove the bottom pad. I was trying to fit in the signal vias and the gnd pad from underneath the IC was obstructing. I mistook the gnd pad as a thermal pad and removed it from the footprint, not realising that it provides gnd reference to the MCU.

[Abidarian]

Hey there guys,
I did a V2 on the board fixing the previous faced issues, and adhering to the design manuals and the section 6.3.8 from the datasheet https://share.google/ZTbqdiSjHi01K7b5x . The is still not perfect and there might be multiple issues. I ran a DRC on this and found no errors. I would appreciate a review and suggestion/ pointers on where I might do things in a better way. Thank You!
Here is the KiCAD project file. This isn't perfect and I am open to suggestions.
https://www.dropbox.com/scl/fi/ebww8nrs9y2mub479a8ma/flip_pendant_v2.zip?rlkey=vz2qk7mp2eregzf99homrwcmn&st=6vt6cggg&dl=0

[unikeyname]

any board layout image?

Yes I am attaching the images with this post. I am sorry if this isn't what you were asking for.
Here is the 3D view of the PCB bottom where all the components are (Attachment Link)

Here is the bottom layer Cu (Attachment Link)

Hi, good to see you are making progress.

Unrelevant but, how do you make `pcbnew` ... tilt like we have a 3D viewport there?

[Abidarian]

any board layout image?

Yes I am attaching the images with this post. I am sorry if this isn't what you were asking for.
Here is the 3D view of the PCB bottom where all the components are (Attachment Link)

Here is the bottom layer Cu (Attachment Link)

Hi, good to see you are making progress.

Unrelevant but, how do you make `pcbnew` ... tilt like we have a 3D viewport there?

I am not sure if I got exactly what you are asking for but I think you wanna know how to render a 3D CAD for the PCB. I am using KiCAD and there is an inbuilt 3D viewport where you can checkout the PCB. You can find more details here
https://www.kicad.org/discover/3dviewer/

[tooki]

any board layout image?

Yes I am attaching the images with this post. I am sorry if this isn't what you were asking for.
Here is the 3D view of the PCB bottom where all the components are (Attachment Link)

Here is the bottom layer Cu (Attachment Link)

I am not sure if I got exactly what you are asking for but I think you wanna know how to render a 3D CAD for the PCB. I am using KiCAD and there is an inbuilt 3D viewport where you can checkout the PCB. You can find more details here
https://www.kicad.org/discover/3dviewer/


Hi, good to see you are making progress.

Unrelevant but, how do you make `pcbnew` ... tilt like we have a 3D viewport there?

You need to add the text of your reply outside the “quote” tags. You can edit your post to fix it.

[unikeyname]

Not that one. I am referring to the method of viewing it in a different projection in `pcbnew`.


EDIT: turns out the render of that images fooled me. From what I looked at, I mistook the visual glitch as an unorthodox projection of PCB while in fact it's just being rendered with the wrong size. Just dont mind me and move on you guys.

EDIT2: Next time you can consider dropping artifacts in somewhere not dropbox. I dont have an account in there so... either github, gitlab or codeberg works for me since KiCad project are pretty much just plain-text data with code like format.

[Abidarian]

any board layout image?

Yes I am attaching the images with this post. I am sorry if this isn't what you were asking for.
Here is the 3D view of the PCB bottom where all the components are (Attachment Link)

Here is the bottom layer Cu (Attachment Link)

I am not sure if I got exactly what you are asking for but I think you wanna know how to render a 3D CAD for the PCB. I am using KiCAD and there is an inbuilt 3D viewport where you can checkout the PCB. You can find more details here
https://www.kicad.org/discover/3dviewer/


Hi, good to see you are making progress.

Unrelevant but, how do you make `pcbnew` ... tilt like we have a 3D viewport there?

You need to add the text of your reply outside the “quote” tags. You can edit your post to fix it.

My bad on that, I have edit the quote tags on the post. Thanks for pointing it out.

[Abidarian]

Oh yes, GitHub would be a great option. I have uploaded the files at this repo https://github.com/abid-sayyad/nia_flip
Thanks for the suggestion unikeyname.