Need help finding high current drain on my PCB

[fcb]

Last test for night, getting frustrated and been working on this all day

Took bad board, and removed:

R15-no change
U5-no change
U4-no change
U2-no change

Device is currently reading 2.38mA.  Should be 1.6µA.   So other than Murata module, all the ICs are removed.  Not sure I'm going to be able to mount them back on successfully.

So logically, the current drain is either coming from the microcontroller on the murata module or the SX1276 on the murata module?  So you pretty much have 4 (2^2) posibilities.

One other thing - modules that exhibit high current when asleep - does this fall in a consistent narrow range?

[Jeroen3]

Could it be ESD damage?

[ogden]

In the list of suspects I would include ceramic (decoupling) capacitors that could receive mechanical stress during manufacturing or use, like those close to mounting screw. Try to replace caps, measure leakage.

[Kasper]

Could it be ESD damage?

I am wondering that too.

sailah, did you recently change offices? Maybe to one with carpet? Or start wearing sweaters?

The people at my work ignored my esd precautions and got away with it until we moved to an office with carpet then 'all of a sudden' boards started dieing.

[Kasper]

Last test for night, getting frustrated and been working on this all day

Took bad board, and removed:

R15-no change
U5-no change
U4-no change
U2-no change

Device is currently reading 2.38mA.  Should be 1.6µA.   So other than Murata module, all the ICs are removed.  Not sure I'm going to be able to mount them back on successfully.

On the plus side, there are less parts to troubleshoot now. Now keep removing until current goes down.  Remove code , caps, transistor. 

I would start with writing minimal code. Just boot up and put everything to sleep.

You might get to a point where all you have on the board is that module and then you can try digging into the module, swapping modules and or improving the FW.

[sailah]

One other thing - modules that exhibit high current when asleep - does this fall in a consistent narrow range?

"we" meaning instructions from my new EEVBlog friend, have been to modify the board with some small caps and ground wires to fix layout issues.  With this device we are datalogging current but from all accounts, this bad device has had any layout defects fixed.

The answer to your question, if you are asking if the current range is fairly narrow on bad devices, I would say yes.  Kinda.  However we are seeing some that "climb" from say 160µA to 500µA and then go back down.  All with no input.  And they should be very low, say 1.6µA for 59 minutes save for watchdog spike.

We have a device now that is fairly unstable.

But generally bad devices will go to 2.3mA, for example, and stay there.  They will join LoRa just fine and have the big pulse during TX as any othe rdevice would.  Just background current is very high and consistent.

Could it be ESD damage?

Anything is possible at this point.  I assemble these on an ESD mat that is grounded but I can't say I'm super careful with them.  I have a slate floor in my office, no rugs or anything.  I ground my arms on the mat.  I used to wear an ESD wristband but I haven't in awhile.

In the list of suspects I would include ceramic (decoupling) capacitors that could receive mechanical stress during manufacturing or use, like those close to mounting screw. Try to replace caps, measure leakage.

Caps seem to be the 1 common link between fixing devices.  For whatever reason by soldering/desoldering caps, sometimes a board will fix itself.

I had a device behaving poorly last night with high current (2.3mA steady).  Had done nothing to it other than let it sit.  Tested today, 1.6µA.  WTF.

Could it be ESD damage?

I am wondering that too.

sailah, did you recently change offices? Maybe to one with carpet? Or start wearing sweaters?

The people at my work ignored my esd precautions and got away with it until we moved to an office with carpet then 'all of a sudden' boards started dieing.

No my office has always been the same, stone floors, no carpet no static shocks ever.  We have these made by a good CM in New Hampshire from domestic PCBs.  They ship in ESD bags.  I assemble into die cast housings and flash them on an aluminum chassis programming jig based on STM32 discovery board with pneumatic pogo pins.  Always been that way.

One thing of note, since the piezo was brought up, we bond these in place with a special CNA glue.  And clamp them in pneumatic fixture with a small air piston holding a rubber foot.  It's about the same pressure as a steady thumb would exert but it's fairly flat.

Last test for night, getting frustrated and been working on this all day

Took bad board, and removed:

R15-no change
U5-no change
U4-no change
U2-no change

Device is currently reading 2.38mA.  Should be 1.6µA.   So other than Murata module, all the ICs are removed.  Not sure I'm going to be able to mount them back on successfully.

On the plus side, there are less parts to troubleshoot now. Now keep removing until current goes down.  Remove code , caps, transistor. 

I would start with writing minimal code. Just boot up and put everything to sleep.

You might get to a point where all you have on the board is that module and then you can try digging into the module, swapping modules and or improving the FW.

We are working on that now. Hopefully the device we have is a good known state and can debug from there.

I'm learning drinking from a fire hose so thank you to everyone, it is sincerely appreciated.

[jmelson]

Caps seem to be the 1 common link between fixing devices.  For whatever reason by soldering/desoldering caps, sometimes a board will fix itself.

I had a device behaving poorly last night with high current (2.3mA steady).  Had done nothing to it other than let it sit.  Tested today, 1.6µA.  WTF.

OHHH!  I wonder if it is some contamination / solder flux under parts?  Maybe take a misbehaving board and put in an oven at 50 - 70 C for an hour, and see if the problem goes away.  The heat will dry out the flux residue.  Do you wash the boards after soldering?
Maybe you are not getting them completely clean, and solvent is left under parts.

Jon

[sailah]

These are made by my contact manufacturer in the US   I assume they do good work but I'm willing to look at anything at this point.

[Psi]

Does this PCB get flexed at all?

[Siwastaja]

Just keep removing parts. You must be quite close, seeing you have removed many suspects already.

Guessing is sometimes OK but there are still too many guesses to test out. Just keep narrowing it down, you'll find it that way.

[fcb]

Without sounding like a stuck record, you need to measure the current drawn by the microcontroller and the current drawn by the SX1276 (Lora IC).
Find and break a trace for the power to each and  insert a 1ohm resistor, measure voltage drop across resistor under ‘fault’ conditions.
This will tell you where to look.

[Siwastaja]

Without sounding like a stuck record, you need to measure the current drawn by the microcontroller and the current drawn by the SX1276 (Lora IC).
Find and break a trace for the power to each and  insert a 1ohm resistor, measure voltage drop across resistor under ‘fault’ conditions.
This will tell you where to look.

Yes and now this is easier than ever, thanks to many removed parts.

[sailah]

Does this PCB get flexed at all?

No, not really.  It's mounted securely to a heavy die cast housing made specifically for it.  Maybe some of the screws might induce a flex to the board but seems improbable.  These devices I have have been sitting in a box not moving along with 100's of other for my customer.

Just keep removing parts. You must be quite close, seeing you have removed many suspects already.

Guessing is sometimes OK but there are still too many guesses to test out. Just keep narrowing it down, you'll find it that way.

I removed all ICs from that one board.  I don't think I could likely put them back on. 

Here's where a quick test went on that board. 

Started with just Murata module...3.6mA constant current
Removed piezo leads...no change
Removed C15 under power, blew fuse, removed C15 and blown fuse, jumpered fuse pads...1.01mA
Removed C14, jumpered fuse pads...550µA
Removed C13, jumpered fuse pads...2µA

Put on keysight and datalogged it.  This is what it looks like.  Other than the join not working because no EEPROM for keys, it worked OK.  This is what a good device should look like at rest.  And I should note that removing this cap array on previous devices has triggered this same result.  So for whatever reason, removing C13-C15 is what "fixes" it in many cases.  C13 being the bulk 10µF cap makes the most difference.

[ogden]

Removed C15 under power, blew fuse, removed C15 and blown fuse, jumpered fuse pads...1.01mA

This could be event that restarted IC - actual consumer. First thing you shall do - write "blank" firmware that does nothing besides low power sleep function, flash all suspect PCBs and see it fixes problem or not. Sorry if this is already said because TLDR, did not read whole thread.

[sailah]

I don't want to prematurely say we found the issue but I think we are fairly sure.

After all this messing around with ICs, floating pins etc...

I took the board as described above and removed caps, sure enough it went down.  But I have seen this before.  Put caps back on, still at ideal current.

So then we took a device that had never had cap array reflowed or soldered.  As in direct from my CM.  Cut the traces for the 3 caps, 1 at a time instead of heat.

When I cut C13, the current went down to 1.6µA.

We then powered up just C13 on the PCB and it was drawing high current.

I think we've repeated this a couple times but I want to try it on a fresh device that has never had the cover removed since it died.  I think that will validate it.

Now the question is, if this is true, how to avoid it?  I will certainly talk to my CM but I want it to be constructive.  Are there ideal caps to be using in the case? 

C13 cap that WAS on all these boards is https://www.mouser.com/ProductDetail/Taiyo-Yuden/TMK212BBJ106MGHT
C13 that I have at my CM for the next round is https://www.mouser.com/ProductDetail/Samsung-Electro-Mechanics/CL21B106KAYQNNE

And I want to say a special thank you to an EEVBlog member, who has chosen to remain anonymous, for literally 100s of emails back and forth and sticking with my stupidity all weekend to track down the issue.  As well as everyone else here who offered their thoughts and read through rambles of a person(me) not even qualified to handle a soldering iron.

THANK YOU ALL.

I will revisit this thread and update if anything important comes along.

[Kasper]

Great to hear the good news! Good job sticking with it!

Ceramic caps are fragile and susceptible to problems due to board flex. I am not familiar with those problems but some things to consider re board flex:

You have 4 screws and an antenna connector tieing the PCB to the enclosure.  That is 5 mounts.  Not many people follow this rule but the ideal number of mounts is 3.

How flat is the PCB?  If amount of copper is not balanced, the PCB can warp. I think that usually happens during fab, not something that creeps up over time.

How tight is the antenna connector?

What is the tolerance on the hole placement of the antenna connector?

Is there more slop between antenna connector and hole than there is tolerance in hole placement.

Did you change your antenna tightening procedure?

If you draw a line from the antenna connector to the nearest screw, that line goes near the caps in question and it represents an area of possible stress.

[ogden]

I have the + leg of supercap clipped so it's not part of circuit.

I wonder - why you have (expensive) supercap in battery-powered product? [offtopic question, but anyway..]

[sailah]

Great to hear the good news! Good job sticking with it!

Ceramic caps are fragile and susceptible to problems due to board flex. I am not familiar with those problems but some things to consider re board flex:

You have 4 screws and an antenna connector tieing the PCB to the enclosure.  That is 5 mounts.  Not many people follow this rule but the ideal number of mounts is 3.

How flat is the PCB?  If amount of copper is not balanced, the PCB can warp. I think that usually happens during fab, not something that creeps up over time.

How tight is the antenna connector?

What is the tolerance on the hole placement of the antenna connector?

Is there more slop between antenna connector and hole than there is tolerance in hole placement.

Did you change your antenna tightening procedure?

If you draw a line from the antenna connector to the nearest screw, that line goes near the caps in question and it represents an area of possible stress.

How flat is the PCB?  Not sure, feels pretty flat.  Not sure other than putting on surface plate to tell you flatness.

How tight is the antenna connector?  The SMA is soldered to board.  Antenna is put on finger tight but this is done by user.  These monitors affected never left my office.

What is the tolerance on the hole placement of the antenna connector?  Good observation.  Tighter than I would like.  I'm having the die casting retooled in the US from china right now.  I specifically asked for 0.5mm larger hole for this reason.

Is there more slop between antenna connector and hole than there is tolerance in hole placement. There is very little clearance.  This was my fault as I wanted to be able for this housing to pass IP67 and figured it would be good practice to have close tolerance.  Never in my mind imagined this could potentially be an issue.

Did you change your antenna tightening procedure?  This is done by user to big variable.  However all these devices in question (but not all failures in total, we have had devices fail in field) have never had an antenna put on or left my office.  They were all installed in housings though.

If you draw a line from the antenna connector to the nearest screw, that line goes near the caps in question and it represents an area of possible stress.  Yeah starting to see all these things.


I don't have any devices that I have programmed outside of the housing so I can't say for sure it's the cause.

Wondering if soft caps would be a better idea to get me though the next round until I respin the board.  I've had multiple people tell me that my layout has room for improvement as well as a few other upgrades to make the device more reliable.  I would like to incorporate all these into next version.

I have the + leg of supercap clipped so it's not part of circuit.

I wonder - why you have (expensive) supercap in battery-powered product? [offtopic question, but anyway..]

We tried a lot of cells with regards to pulse current.  In Gen 1 of this product we used a Microchip RN2903 LoRa radio with a CR123 which could handle the pulse.

When we switched to Murata module, we opted for a Saft LS14500 due to ATEX (explosive environments) requirements as well as capacity, life.  This cell could not handle the pulse of LoRa radio and radio would reset to to voltage sag.  Saft told us we need a supercap after we provided datalogs.


The 0.47F AVX cap was recommended for a lot of reasons in that it had lowest leakage and was already built for HazLoc.  Whether it is overkill or too big/small I can't comment.  What I can say is that it works really well in reducing the pulsing of the LoRa TX.  In the big picture it's a small overall cost and really helps us get every last bit out of battery with a very low leakage. 

My goal is not to have the cheapest BOM, especially with some of the clients we have.  It's been a very interesting ride in this PCB journey.  5 years ago I didn't even know what a capacitor or IoT was.  Like I said, sales guy.  Now I know just enough to be dangerous...

https://www.mouser.com/ProductDetail/Kyocera-AVX/SCMR14C474PSBA0H

[ogden]

How flat is the PCB?  Not sure, feels pretty flat.  Not sure other than putting on surface plate to tell you flatness.

Usually it is about errors in different (X/Y) planes. In your case you shall be checking against antenna socket contact with enclosure. [edit] I see there is no nut on the antenna socket. Meaning PCB is pulled as strong as consumer pleases to tighten antenna.

When we switched to Murata module, we opted for a Saft LS14500 due to ATEX (explosive environments) requirements as well as capacity, life.  This cell could not handle the pulse of LoRa radio and radio would reset to to voltage sag.  Saft told us we need a supercap after we provided datalogs.

That sounds like overkill advice, especially considering battery used. I would suggest to test electrolyte cap in place of supercap, start with 1000uF or so. You can save some money.

[sailah]

Running a final test to confirm on an untouched device

Grabbed a failed monitor, battery was 0.016V.

Tested monitor just by powering up.  Joined network as expected, current was high but couldn't see precisely because supercap was still on.  Removed 1 leg of supercap.  Current was 55µA and jumped up to 100µA, very unstable.  (Picture 1)

Then I cut the trace between C13 & C14, verified isolation.  Current drops to 1.5µA as it should be.  Very stable, very narrow range.  Just the spikes from the watchdog.  I've seen this power profile so many times I could draw it from memory.  Picture 2

Then I replaced C13 with a new cap from Samsung, same 10µF, 25VDC, but X7R.  Not sure if this matters or if it was bad soldering.  Soldered in a jumper leg across VCC from C13 to C14 to re-establish the trace.  Looks exactly as picture 2 does so no need to show it again.  Current 1.5µA

So then I resoldered AVX supercap, charged it up to avoid blowing fuse.

Running a 24 hour datalog to compare current consumption to what we have see in many good devices.  Should use approx 5-6J daily.  My test will be a little skewed as the cap chargeup takes awhile to stabilize.  I'll leave it on logger for 24 hours anyways but plan to run another log tomorrow just to be sure.

[Manul]

Something was really going wrong, that such high percentage of devices developed mlcc capacitor failure. If that was caused by soldering / mechanical stress, soft / flexible termination capacitors should reduce failure rate by at least 1 or 2 orders of magnitude.

That sounds like overkill advice, especially considering battery used. I would suggest to test electrolyte cap in place of supercap, start with 1000uF or so. You can save some money.

I think, the supercap was needed because of that diode voltage dropper, made of 3 diodes. At very low (idle) currents, they drop very little, basically "leak" and charge the capacitor very near battery voltage. At higher current, the voltage drop through these diodes immediately becomes much much higher, up to maybe 1V - 1.5V, so that is the reason why small electrolytic was not enough. It is more like design flaw. 3 diodes are used to drop enough voltage at idling 1.6uA, but they are also very high impedance at that point.

[sailah]

There are some design considerations that forced decisions based on ATEX Class 1 Div 1 (HazLoc) certifications that we were forced to use for certification reasons.  They might not make sense from an electronics perspective but they were needed for protection against spark/thermal rise.  I'm only echoing what I had to go through but I don't have any comments on if it could be a better design, of course it could.  But here we are.

2 hours into datalog and everything looks great.  Our device samples 30x last minute of every hour and uses a decent amount of energy, otherwise asleep.  Every 8 hours it sends a LoRa Tx.  From what I can tell it's all working perfect.

I'll update after I speak to CM tomorrow and get their thoughts about capacitor/solder issue.

Thx

[Kasper]

For next version, I would look into different ways to attach that antenna. I'm sure lots of people have faced this problem so there are probably lots of good ideas out there for it. 

For starters, you should be able to get connectors that can be fastened to the enclosure and soldered to the PCB. That is what I assumed you had.  I think it'd be ideal to fasten it first then solder it but I'm not familiar with that.

What am a little familiar with is fasten connector to enclosure and have coax cable between connector and PCB. That's a bit of a pain though because it means 4 connectors and a cable instead of 1 connector. And you have to connect the cable without twisting it too much or bending it too sharply. And if you damage the cable it might be hard to tell, might just have less range than normal.

[ogden]

I think, the supercap was needed because of that diode voltage dropper

Oh, right. Low quiescient current LDO shall be used instead in such kind of app.

[Manul]

Wanted to share a really nice, quite extensive paper focused on mlcc capacitor cracks. Some really interesting data and insights there.

https://ntrs.nasa.gov/api/citations/20190001592/downloads/20190001592.pdf