Need help finding high current drain on my PCB

[sailah]

Hi, thanks for reading my post.  First off I don't know enough to likely pose a decent question but I hope to learn so that I know what to ask an expert.  My real goal here is to find an expert that I can engage with to help finally hammer this issue and put it to rest.  My developer who built the circuit and firmware has tried to find the issue but it keeps happening.  My US contract manufacturer says there are no hardware issues that are obvious.

We build a simple PCB running the Murata CMWX1ZZABZ-078 LoRaWAN module which uses the STM32L0 MCU.  We followed the reference design.  We've now built 2K of these and they mostly work excellent.

However an alarmingly high number of devices (1-5%) will exhibit very high current drain and flatten a 14500 lithium primary cell in 1-5 months.  We've done extensive power profiling of these on our Keysight N6705C and a good PCB will show ~12µA continuous and a life of 13+years.  Devices that are behaving poorly are showing ~450µA drain.

I had purposely flashed 500 of these devices and found ~15 or so that dies prematurely.  Took a device that lasted 132 days and erased flash with STLink.  Reflashed with same firmware and it immediately rejoined the LoRa network as it should.  Left it overnight to stabilize the supercap.

Put on Keysight and datalogged power profile.  Marker 1 is typical, we are sampling 30x in 1 min.  Then it should be a log slope down to next sample at marker 2.  But 15 min in it goes all wonky.  I removed supercap and tested again, seeing 2.77mA resting.  Should be 12µA.

Hoping I can learn a bit more about debugging this.  I can do most things like solder/desolder, power profile, use DMM etc.  But anything above that is greek to me.  I do have an Agilent scope, Siglent VNA with probes, Keysight N6705C with SMU.  And I know what a good device should look like.  If you think you can help me, I'm happy to engage. 

Thx

[sailah]

This is what a good device looks like for 1 hour.  Marker 1 shows our 1 minute sample, then recharging 0.47F supercap until next sample.  Looks like average over that hour is 11.7µA.

We thought it could be inputs on MCU "not pulled up/down" I forget but I believe we addressed that in firmware.

Tried removing power caps 1 at a time but I don't recall that made a difference.  We really thought it was firmware but I have no idea.

[fcb]

Depending on how many channels you have (N6705C has 4 if fully loaded?) can you power the suspects (LAN, MCU, etc..) from seperate channels.  Perhaps prove which part is misbehaving?

How do the suspect units behave at low temperatures compared to the known good units?
Die revisions?
Are boot times IDENTICAL?

[sailah]

Depending on how many channels you have (N6705C has 4 if fully loaded?) can you power the suspects (LAN, MCU, etc..) from seperate channels.  Perhaps prove which part is misbehaving?

How do the suspect units behave at low temperatures compared to the known good units?
Die revisions?
Are boot times IDENTICAL?

I have the N6705C loaded but just 1 SMU (N6781A) for battery drain analysis.  I guess I'll admit I wouldn't know how to power the various circuits separately.  If it involves cutting traces that's probably above my pay grade.

When you say die revisions I assume you are talking about the Murata module?  They were all from the same 1K reel so they should be identical.

I can try boot times and compare.  From memory, and I've looked at a lot of my datalogs, it's pretty darn close but I will check and compare this baddie with a good one to see if boot time is same.

Thx

[Siwastaja]

Do binary search.

Cut power traces in half, measure the halves separately. You don't need many channels to do that. After you found which side is misbehaving, again cut that half and so on.

"Half" not necessarily meaning geometrical sides, but half the components, you know.

Even easier, just remove some parts completely, this might render the unit nonfunctional of course, but this lets you see if the problem disappears.

[sailah]

Interesting I was not expecting a difference but there was.  I took an identical PCB, removed the supercap as I had done with device I am testing.  Erased flash and flashed with same firmware.  It joined the LoRa network just fine.

Here's the bootup profile from the device behaving poorly.  At Marker 2 seeing 469µA, way too high.

Takes about 7 secs between packet sends.  Not sure if this is ideal way to measure but those plateaus are the TX from LoRa and easy to pick markers from.  The small plateau to right of M2 is the RX downlink from LoRa gateway

[sailah]

Here's the good device that I just flashed.  Again no supercap.  Shorter bootup.  At Marker2,  30µA, and after plateau, it's down to 2µA.

[sailah]

Do binary search.

Cut power traces in half, measure the halves separately. You don't need many channels to do that. After you found which side is misbehaving, again cut that half and so on.

"Half" not necessarily meaning geometrical sides, but half the components, you know.

Even easier, just remove some parts completely, this might render the unit nonfunctional of course, but this lets you see if the problem disappears.

Thanks, I think we did that last year when this issue cropped up.  Thought we found something related to a cap but could never repeat it.  What I did was desolder C13, C14, C15 1 at a time to see if it changed it.  It did not.  Although I could try again as my memory is fuzzy from back then.

Attached schematic, the external temp sensor circuit is not populated on any of these boards as well as diode on bottom right.  Capacitor array is what we tried to remove.  I believe I even soldered in different caps from another manufacturer to see if that changed it but I should probably repeat this process.

My developer had not pulled up/down inputs on MCU where accelerometer was involved.  The accelerometer was mounted but not used in firmware.  I think there was concern about resonance so he changed that in firmware.  Issue remains.

[Siwastaja]

So it's not a huge number of parts. Just desolder 2-3 ICs and their related passives. Something that is not fundamental to the most basic operation, so that it can still boot up, maybe with a minor firmware change removing some error check and letting it boot and enter low power mode or whatever it needs for you to be able to observe the issue. Accelerometer could be removed, for example.

[Manul]

I did not read all, but my first impression is that maybe something is floating? I would try like this: make a Vcc/2 divider with two 100k resistors attach a wire to the middle (Vcc/2 voltage) and touch all the pins of every IC. Observe current consumption. It does not take long, and you might find surprises. Perform this on a "good one".

[sailah]

So it's not a huge number of parts. Just desolder 2-3 ICs and their related passives. Something that is not fundamental to the most basic operation, so that it can still boot up, maybe with a minor firmware change removing some error check and letting it boot and enter low power mode or whatever it needs for you to be able to observe the issue. Accelerometer could be removed, for example.

OK I'll ask my developer for some guidance on this.  I can removed things with hot air station but getting them back is not so easy for me.  I barely can get 0805 caps on lol.  I'm a sales guy who happens to need to be involved in this.  I wish it weren't the case but that's life.

I performed another experiment which we tried last year.  Removed all caps C13-15 1 at a time.

Removed C15-No change, background current at 2.77mA.  After first sample, this dropped to 470µA on its own
Removed C14-After join dropped to 300µA
Removed C13-After join about 2.3µA

OK so removing C13 got resting current to where we expect and have observed many times.  I then replaced all the caps after much swearing and sniffing fumes.  Reused C14 but C13 & C15 were different manufacturer.  Back up to 2.36mA.

But something happened with that C13 removal that triggered a major drop in current.  I don't know what it could have been but the Keysight rarely lies.

So then I removed all 3 caps again.  Back to 2.36mA

[sailah]

I did not read all, but my first impression is that maybe something is floating? I would try like this: make a Vcc/2 divider with two 100k resistors attach a wire to the middle (Vcc/2 voltage) and touch all the pins of every IC. Observe current consumption. It does not take long, and you might find surprises. Perform this on a "good one".

OK I will attempt this as I enjoy learning.  Might need to watch a tutorial as my understanding of this concept is very low admittedly.

If there was a scale of EE understanding from 0-100, I'm about a 2.

[Manul]

OK I will attempt this as I enjoy learning.  Might need to watch a tutorial as my understanding of this concept is very low admittedly.

If there was a scale of EE understanding from 0-100, I'm about a 2.

To say it simple, in digital circuits, every digital line (even unused inputs) should be either at Vcc or Gnd voltage. If voltage is somewhere in the middle - there will be increased current consumption. This is why unused inputs should be connected to Gnd, Vcc (directly or through pulldown/pullup resistors) or configured as outputs (if it's MCU). If left on their own (floating), these input may randomly develop various voltage levels and increase current consumption. Even oscillate in some cases. The test with Vcc/2 divider tries to pull pins to the middle voltage, this should not be succesful if everything is designed/configured properly. But if succesful, you should see an increased consumption and this would mean that it is a misconfigured digital line. I tried to explain as simple as possible. Note that you will see slight increase in consumption because of divider itself, but just ignore it, look for increase which can not be accounted by divider current.

[fcb]

So there is a difference between a good and bad boot-up....  I'd probably look very closely at the firmware at this stage.

If you can drive or toggle a pin when you reach certain places in the boot code, then you should be able to see which routine is not performing correctly (like perhaps not setting the clock tree right, or configuration of a peripheral incorrectly).  It maybe that you are relying on a default state of some register somewhere, and that the default state has changed for some reason (die revision, perhaps on SX1276?).

Just because the module came from the same batch, doesn't mean that the components that go into the module also came from same batch.

[fcb]

Long shot. Also, try removing R14.

That 10K thermistor (assume NTC) getting grounded by a misconfigured analog pin during sleep might warm up a bit (dropping impedance to say 7.2K...) and perhaps contribute another 456uA of load.

If there is enough time between thermistor samples (say a couple of seconds), then you could measure the drop across this in-circuit.  Any voltage across R14 when not sampling is not correct.

[sailah]

Thanks for everything so far.  We didn't get to voltage divider but we did probe around with a scope looking for oscillations.

What was weird was that after probing a bunch of stuff (and seeing nothing), my developer said leave it alone during that 1 min of sampling every hour.  Sure enough, right after that hour and 30 samples, it drops down to 1.5µA.  No matter what I did I couldn't get it back to a fault state.

I have a few more to try and will try to replicate the failure.  The plan is Friday to do a datalog session and leave the device in the programmer hooked to pogo pins so we can reflash it without touching it.  We are going to rebuild firmware from the ground up.  First firmware will just be sleep state etc and keep compiling until we can can trigger it.

After watching the device fix itself after a 1 hour session, it def feels like a firmware thing.  We talked about it at length and he said the unused pins are all pulled up in this latest firmware.  There are some pins from the STM32 that we don't have pinned out on the Murata module.  He thinks there might be something there so we will investigate what is going on with the unused pins on STM32.

Will keep at it, thanks again for reading and replying, feels like I've been alone on this for way too long.  If anyone wants to get deeper involved as a consulting thing and feels like this is in their wheelhouse, happy to engage to.

[Manul]

You will likely not see oscillations with a scope in case of floating digital line. The reason is that probe itself will likely be enough load to pull it down and stop it.

Again, I'm not saying that floating pins is for sure your problem, but seems worth checking. You will not believe how strange it may act. It might be totaly unpredictable, there are leakage currents inside IC, also PCB level leakage currents (like someones fingerprint, or soldering flux leftovers) it may react to humidity, external EMI and so on. So the line may randomly transition in logic states, hang in the middle causing increased consumption, or randomly wake up MCU if it has interrupt attached. So it would theoretically fit your description, that some boards are affected and some not.

Unless your firmware is very complex, it sounds strange that 5% is affected and others not. Typicall, firmware is quite deterministic.

I quickly looked your schematic. Noticed, that you do not have pullups on CS lines. Generally, you should have, because while MCU is in reset, they become undefined (this is not likely to be related to your problem). What can be related to your problem is STM32 MISO line, because if no CS line is activated (bus is not used), then it is tri-stated (floating). So check that you have pullup enabled on this one (in firmware).

Also, I see you have interrupts attached to accelerometer? Maybe it causes MCU to wake up?

[Kasper]

When things happen randomly, it reminds me of when I used to work somewhere that sold wireless doorbells. One of them would ring randomly, not often but sometimes in the middle of the night. After they returned it, I found a pullup resistor on the button net that wasn't soldered properly.

[Siwastaja]

I have to remind about a case where in a batch of 30 assembled boards, IIRC in two units the accelerometer was not initializing. In all the others, never any issue whatsoever.

Must be soldering issue by the (Chinese) CM, right?

Wrong. Software issue on my side. Initialization didn't exactly follow the requirements in the datasheet, which made two of the exact same accelerometers from the same batch not initialize, all others just fine.

I don't remember what it was exactly but it was not a bus timing violation (in which case it would be obvious that some units work with marginal timing). It was really weird and totally unexpected.

[sailah]

Thx, we are currently working on this in parallel.  I have a coder I brought on to help spot any issues and build firmware back from scratch.  Doing this right now and monitoring changes in Keysight.  Starting at the most basic STM32 firmware and going in stages.

Also have original developer of firmware doing the same thing on Friday.  I've offered a cash prize to see if someone comes up with the fix first, hopefully a bit of friendly competition.

I've mentioned the suggestions here from helpful members, thank you.  I will report back on what we find.  If it's one of the suggestions made here, I'll be sure to be in touch as you win a prize too   If you want to be more involved I'm open to proposals...

These boards have always been made in US.  I use domestic PCBs and PCB-A work is done by my CM in New Hampshire.

As always, thank you, this has been helpful to share the struggles as we get our product debugged.

[Doctorandus_P]

Another cause of weird issues is using an SPI bus without pullup resistor.
If the chip select is floating during reset and initialisation of the microcontroller, then external devices may receive faulty data and end up in weird states.

[Manul]

Just remember, whatever you do (for example, rewriting firmware), try to find proof of what really was a problem. Sometimes it happens, that the issue is fixed, but the doubt remains (that is really annoying). Or even worse - issue randomly apears again after some time, because it was not really fixed.

Also, completely rewriting the firmware introduces risk of new bugs. Be careful, you don't want your devices returning back to you again because of some new issue. Remember, 95% of them worked well.

[Siwastaja]

Yes, I'd say it even more strongly, from experience, that if you seemingly "fix" the issue without completely proving, waterproof, what the cause really was, it's almost 100% guaranteed the bug will come back at the worst possible time.

At very least you need to prove where the bug is, but that is already risky because there is a risk of misidentification of the source when the mechanisms is unknown.

[fcb]

+1 for what others have said!

Absolutely don’t start re-writing code - you need to identify and prove the mechanism that causes the problem.

 

[Siwastaja]

Rewrite is OK if the whole thing is a nest of problems like this. If it's otherwise working OK and this is the only major issue - don't even consider rewrite.