SAMD21G18 Hardfault Debugging

[oldfortune_]

Hi All,

I've been working on some lighting objects recently, and I released a limited batch of 10 for testing last year. All of the units are fine, however 1 has stopped working after 8 months of continuous use. I've got the PCB back in the lab to try to find the fault, and I'm not having much luck unfortunately, so I thought I'd reach out to the forum

So far I have narrowed the issue down to a fault in the microcontroller, but here is a little info on the PCB, and what I have tested so far:

- Atsamd21g18
- potentiometer inputs
- gpio -> mosfet to 24V LED strip
- piezo buzzer
- CUI buck converter module 24 -> 3.3V

Evaluation:
- All the hardware works fine, after swapping the microcontroller all the functionality is there.
- After digging a little with the faulty micro it seems that the program gets held up because of a hardware fault.
- See attached images for the register values at that point in the flash as compared to a reference micro.
- There is a hardfault listed on the IPSR
- I read back the flash from a freshly flashed micro, and the faulty micro, and found that there were a few differences (this feels like a red herring though)

Questions:
- Any idea how to identify what the IPSR = 0x003 is.
- Any ideas what might have happened, I suspect some sort of voltage spike damage.
- is it possible to cause this damage with faulty firmware?

Hope thats enough info to have some initial insights. Happy to upload schematics also if thats helpful.

Cheers, Sean

[ataradov]

The way to identify a HF is to set a breakpoint at the beginning of the handler before any other registers are modified.  And then look at the top of the stack. There will be a program counter at the time of the fault and a few other useful registers. Then you can disassemble the code around the saved PC and in combination with the saved registers, you will know what caused the fault.

But this will not tell you what exactly happened on a large scale. You may not even supposed to be running that code and the actual fault is in some other place. This is not a trivial thing to investigate in general.

You say you read the firmware from the faulty MCU and it has bit differences compared to the expected image? How it that a red herring? Looks like the reason for the fault, or at least something that needs investigation.

Can you show example of those differences?

[oldfortune_]

Thanks for the response!

See attached a couple of screen grabs of a hex compare of the flash memory. Faulty MCU as compare to the fresh out of the box working MCU.

I think the fact that a bunch of the registers are set to 0xFFFF FFFF seems a little strange in the previous micro status screen grabs.

Cheers,
Sean.

[ataradov]

Yeah, those are bit errors. They are 100% the reason for the faults. You either have a faulty MCU or something else is wrong on the board and one MCU is a bit more sensitive.

Double check VDDCORE and see that it is at 1.2 V and clean (no ripple or other noise).

And you can see that bit 3 (value 4) is the reason for the difference in all cases and it is stuck at 1. This is how flash issues typically present themselves.

Furthermore, you can try to do multiple reads with power cycles between them and you are likely to observe that those bit errors move around and sometimes affected bytes recover. This is also a sign of marginal flash performance.

[oldfortune_]

Good spot! Is it possible that this sort of flash fragility can occur from just 8 months of use? That seems way too short. Or does this massively depend on the power supply stability etc.

[ataradov]

It depends. If your VDDCORE is noisy or you have massive EMI issues in your design, then it can happen even faster.

At the same time, you can't exclude the actual device defect. This happens too and the only way to find out is to file a support ticket and eventually submit the device for a failure analysis.

[oldfortune_]

Here's a couple of scope grabs of VDDCORE: looks to be sitting around 1.26V avg, any thoughts on that? The PSU is just 24V DC in, and a CUI drop in Buck to get a 3.3V supply rail.

[ataradov]

This is really bad. First of all, make sure this is not an observation issue. Instead of the ground alligator clip, use the ground spring contact on the probe. This will minimize the loop area and may eliminate some noise. If the noise is still present, you will need to investigate it.

And then, use a multimeter to measure the voltage on the pin more accurately. It must be ~1.2 V. Multimeter will average noise and show you a real DC value.

Can you show your VDDCORE part of the schematic?

[oldfortune_]

Ok so multimeter measurement is at 1.23V. And see attached the samd21g18 schematic section as well as the relevant board layout.

The buck module is the following.
https://www.mouser.co.uk/ProductDetail/CUI-Inc/PXO7803-500-M-TR?qs=By6Nw2ByBD3KcmSyN1WOOQ%3D%3D

[ataradov]

This is not nearly enough decoupling. Look at the schematic checklist of the datasheet.  VDDCORE capacitor must be 1 uF. Ideally you want 1 uF || 100 nF.

I would also add 1 uF in parallel with C9. Whether you really need it depends in the rest of the board layout, but it will not hurt in any case.

And you need 100 nF for each VDDIO pin.

VDDCORE is the biggest issue, and needs to be addressed immediately. The rest are just good practice.

[oldfortune_]

Ok cool, and so you think that the VDDCORE insufficient decoupling could cause these permanent flash memory failures?

What would an acceptable ripple on VDDCORE be, with correct probing?

Thanks for all the answers

[ataradov]

VDDCORE noise can cause permanent flash damage. Whether this noise is due to insufficient decoupling or from external factors, does not matter.

Ideally you want to keep it below 10 mVpp, but I would really start looking into things at 20 mVpp and above.

[oldfortune_]

Ok thanks for taking a look! I think I pulled a reference for the decoupling from an open source design, which I now see a couple of others have pointed out failure due to bad VDDCORE decoupling!

Good tip to be really careful with that one!

Cheers,
Sean.