Need help finding high current drain on my PCB

[sailah]

OK thanks this is helpful.  I wasn't thinking we were rewriting the firmware from scratch.  I think the plan was to put firmware on the STM32 to just force it to deep sleep and check current.  Nothing else.  If that works, add the next bit of firmware, get LoRa working, add our application etc.  Hopefully along the way the problem arises and we can then figure out what in the code caused this.  And then correct original firmware.

I really need a "fishing guide" here, someone who can help advise on the plan to debug.  I'm the sales person, this is all way above my pay grade.  Doing the best i can to get this back on track.

It does feel like something simple as we've had devices in the field for years that work perfectly.  Our application works awesome.  Just when the battery dies it doesn't work so awesome

I know I've mentioned this before but if this is something anyone here is willing to help with, I do have more projects related to this firmware that I will be needing ongoing help with.  Happy to have those conversations offline.

[Siwastaja]

Yes, I agree it's a really good idea to write super minimalistic test firmwares to reproduce HW problems. I do that all the time, and often I have some initial "HW test firmware" which I can always get back into even if not used in "normal" development.

[alanambrose]

>>> But 15 min in it goes all wonky.

Well a start would be to see what's going on at the 15 min mark. Can you do / switch on some software logging (hopefully the problem doesn't go away when you do this) and see what happens in that timescale. Is it always 15 mins?

Alan

[fcb]

As you are using lorawan, examine how your receiver is setup.

There is a remote possibility that the unit is waiting for some sort of delayed reply packet from the lorawan network before it goes back to sleep.

[sailah]

>>> But 15 min in it goes all wonky.

Well a start would be to see what's going on at the 15 min mark. Can you do / switch on some software logging (hopefully the problem doesn't go away when you do this) and see what happens in that timescale. Is it always 15 mins?

Alan

Don't really know what kind of software I could hook up.  No I would say the wonky behavior is hard to predict.  Last year we had some devices that went sorta nuts.  Was leaving them on Keysight for hours and it would be fine then big spike into the 100s of mA.  Caps were getting hot.  That's when we thought we had a hardware problem but we could never replicate that.

Working with super nice member here probing digital lines at present.  Thanks to everyone for their thoughts and help, it is appreciated.

[sailah]

As you are using lorawan, examine how your receiver is setup.

There is a remote possibility that the unit is waiting for some sort of delayed reply packet from the lorawan network before it goes back to sleep.

I think this is a good thought but seems unlikely.  We are using stock Murata Semtech ST stack to control LoRaWAN to 1.0.3 specs.  I can see the packets come through on our LoRa Network server and see the ACKs on both device using Putty and online.

When this draw is present, it's on the entire time as a increase in base current.  Over top of that on Keysight I can see the normal activity being performed regarding LoRa activity.

[mikeselectricstuff]

  We didn't get to voltage divider

Do this before you do ANYTHING else. It is quick and simple to do, and there is a reasonable chance the issue is something floating. 

Another thing I'd look at is whether there are any hardware registers that are undefined on powerup, and not initialised by your code.

Read the device errata - things happening "sometimes" are the sort of issues that make it through to production

Put a resistor in series with the Lora module's supply to see if it is the module or your MCU drawing the current ( but keep  in mind the current may be flowing through IO lines).

Do you get consistent behaviour between units ? i.e. does one board always draw excess power and another not ?
If so, swap the lora modules over and see what changes.

Have you checked that it's not a leaky supercap ?
Accelerometers typically have register configs to select opemn-drain or push-pull interrupt lines - check this is being set up OK

Try powering up a bad board at high and low ambient temps ( hairdryer, freezer spray). This might show anything where there is dependency on some analogue value. e.g. an RC oscillator somewhere that interacts with code execution speed.

BTW why are you using Q2 instead of just using the IO pin direct to switch the thermistor ?

[mikeselectricstuff]

And as others have said, until you can do something that will replicate the issue to make a "good" board behave badly, you can't be sure you've fixed it, or that units in the field won't suddenly develop the issue.

May be hard to get through to a big company, but definitely worth talking to Murata about  their module - it could be a known issue. I had a similar problem with some 868MHz RF modules a while ago, where they wouldn't always wake up, fortunately it was a small company and they confirmed it was a known bug and reprogrammed the modules with later FW.

[sailah]

  We didn't get to voltage divider

Do this before you do ANYTHING else. It is quick and simple to do, and there is a reasonable chance the issue is something floating. 

Another thing I'd look at is whether there are any hardware registers that are undefined on powerup, and not initialised by your code.

Read the device errata - things happening "sometimes" are the sort of issues that make it through to production

Put a resistor in series with the Lora module's supply to see if it is the module or your MCU drawing the current ( but keep  in mind the current may be flowing through IO lines).

Do you get consistent behaviour between units ? i.e. does one board always draw excess power and another not ?
If so, swap the lora modules over and see what changes.

Have you checked that it's not a leaky supercap ?
Accelerometers typically have register configs to select opemn-drain or push-pull interrupt lines - check this is being set up OK

Try powering up a bad board at high and low ambient temps ( hairdryer, freezer spray). This might show anything where there is dependency on some analogue value. e.g. an RC oscillator somewhere that interacts with code execution speed.

BTW why are you using Q2 instead of just using the IO pin direct to switch the thermistor ?

Most of this is above my head.  I'm literally working right now with a super generous EEV member doing exactly voltage divider testing as requested.  I don't know enough to interpret the results but hopefully he does and is willing to share his conclusions.

I will share your other points with my developer when I meet Friday AM.

I don't think it's a leaky cap.  We routinely remove those for testing with Keysight and leaking current is still there with it removed on bad devices so it's not cap.

I can power up boards in high low temps very easily.  Will report back.  Right now I have a know good board and bad board on bench and we are testing them both.  OK member is telling me what to do am I'm following instructions...

I don't know the answer to your Q2 questions sorry.

We are in direct contact with Murata and they are willing to help.  We've had their device FCC certified and generally they have been very helpful.  In fact my rep just had me fill out a few pages of questions that they asked.

Thank you for your comments, they are much appreciated.

Once we get to the finish line I hope to share the results.

[sailah]

Thanks to an extremely generous forum member who spent a few hours with me via email doing testing, these are the results.  They wish to remain anonymous so I'll call them my results but you know what I mean...

Hope to take this testing into my Friday meeting and get an aha moment.  If not I will bring them to Murata (who has their own line of questions) and see if we can get anywhere.  I did ask ST for some help and have a good relationship there as I buy a lot of ST stuff for my boards.

1. STM32_SWO on P5 goes nowhere (not important);
2. STM32_USART1_RX and STM32_USART1_TX are floating. If UART is not used (and it is clearly not used - not initialized), both of these pins should have pullups enabled;
3. STM32_SWCLK and STM32_SWDIO are floating. Generally these should have pullups on the PCB.
4. There is a lack of bypass capacitors. Especially for U4 and U5. There should be capacitors very near U4 and U5. Having no capacitors might cause soft or hard failures of IC including data corruption, undefined states, latch up or even complete failure due to exceeding maximum voltage ratings (voltage transients caused by excessive ringing).
5. The SPI data bus STM32_SPI_MISO line stays hard pulled (high), while both of slaves (U4 and U5) have CS lines high. Either this is firmware "feature" or some unknown failure. Generally, unused SPI bus should have tri-stated MISO line (with pullup).

[Manul]

It is somewhat strange if SWCLK and SWDIO are floating. I think most stm32 set them internally SWCLK - pulldown, SWDIO - pullup. Maybe it is because of sleep state? Or something misconfigured? Definetly not encouraging to see them float. You should search for firmware misconfiguration or pull them externally and retest. Same for UART, fix that.

Bypassing problems is another can of worms. You should probably bodge some caps if they are missing and retest.

[sailah]

Spent a number of hours testing this morning with developer.

We managed to isolate 5 devices running same firmware, 1 runs perfect and has resting current of 1.6µA.  The others have various resting currents of 180µA-4mA.  In some cases the current is very stable and high, like 2.3mA.  In other cases it will start fairly low (~20µA) and then ramp up to 180µA and continue to show instability.

We did find a couple anomalies.  Pin #3 MAX9814_MICOUT is the output of the audio amplifier.  On a couple devices we were seeing 180-220mV on this pin with picoscope.  It did not correlate with high current but it should have been 0.

In every case, each device had its supercap removed and all joined the LoRa network without issue.

My desk is turning into a war zone here

Will entertain serious cash prize (not sure if this is verboten is so please forgive me) or proposals if anyone wants to remote debug and help me advance closer to solution

[Kasper]

Pretty nice setup for 'just a sales guy'!

Can you share pics of the PCB?  I'd like to recommend some mods to figure out which part is drawing the extra current but that will be easier if we can see what you've got.

[sailah]

Haha, yeah just a sales guy I claim no other skills.  Here's one of my baddies.  I have the + leg of supercap clipped so it's not part of circuit.

Here's a pic of PCB, let me know if you need any other details, I think schematic has already been posted.

Thx

[mikeselectricstuff]

We managed to isolate 5 devices running same firmware, 1 runs perfect and has resting current of 1.6µA.  The others have various resting currents of 180µA-4mA.  In some cases the current is very stable and high, like 2.3mA.  In other cases it will start fairly low (~20µA) and then ramp up to 180µA and continue to show instability.

That sounds a lot like a floaty pin. see if it's affected by a mild static field, e.g. wave a charged plastic comb near the PCB and see if it affects the current draw. If it does then it's almost certainly a floating pin

[Kasper]

We managed to isolate 5 devices running same firmware, 1 runs perfect and has resting current of 1.6µA.  The others have various resting currents of 180µA-4mA.  In some cases the current is very stable and high, like 2.3mA.  In other cases it will start fairly low (~20µA) and then ramp up to 180µA and continue to show instability.

That sounds a lot like a floaty pin. see if it's affected by a mild static field, e.g. wave a charged plastic comb near the PCB and see if it affects the current draw. If it does then it's almost certainly a floating pin

Great idea.

[Kasper]

Haha, yeah just a sales guy I claim no other skills.  Here's one of my baddies.  I have the + leg of supercap clipped so it's not part of circuit.

Here's a pic of PCB, let me know if you need any other details, I think schematic has already been posted.

Thx

Of course it's a black PCB    Green is better, it's easier to see the tracks on it and I might recommend cutting power traces but instead, you can remove parts.

I would try removing these parts from a bad board and see if it helps.  If it does, double check by putting them back on and retest.  Note the FW may need update to make it ignore the missing parts.

R14
U5
U2
U4

If that soldering is beyond you, try a local repair shop, even the little phone repair shops sometimes have people with good skills and equipment.

[sailah]

We managed to isolate 5 devices running same firmware, 1 runs perfect and has resting current of 1.6µA.  The others have various resting currents of 180µA-4mA.  In some cases the current is very stable and high, like 2.3mA.  In other cases it will start fairly low (~20µA) and then ramp up to 180µA and continue to show instability.

That sounds a lot like a floaty pin. see if it's affected by a mild static field, e.g. wave a charged plastic comb near the PCB and see if it affects the current draw. If it does then it's almost certainly a floating pin

Thank you I will def try that.  I mentioned my VCC/2 testing and he said the pins were not floating.  We checked with my Fluke in ammeter mode to ground and looked for an increase in current?  But I have a number of people saying they are floating.  I will run your test tonight and report back.

Haha, yeah just a sales guy I claim no other skills.  Here's one of my baddies.  I have the + leg of supercap clipped so it's not part of circuit.

Here's a pic of PCB, let me know if you need any other details, I think schematic has already been posted.

Thx

Of course it's a black PCB    Green is better, it's easier to see the tracks on it and I might recommend cutting power traces but instead, you can remove parts.

I would try removing these parts from a bad board and see if it helps.  If it does, double check by putting them back on and retest.  Note the FW may need update to make it ignore the missing parts.

R14
U5
U2
U4

If that soldering is beyond you, try a local repair shop, even the little phone repair shops sometimes have people with good skills and equipment.

Haha, yeah but I like black and I picked the color...

I can easily remove those.  I presume you want this done on faulty board while monitoring current. 

R14=thermistor resistor
U5-Microchip EEPROM to store keys
U2-Max mic preamp, needed for firmware but maybe not?  Used at end of every hour
U4-3 axis accelerometer.  Not used in current firmware, so should be shut down.

I can unsolder all of this safely I have a rework station and Hakko iron.  Not sure I can get the acelerometer back on or even the preamp but I can try

Will report back tonight.

Thx

[Kasper]

We managed to isolate 5 devices running same firmware, 1 runs perfect and has resting current of 1.6µA.  The others have various resting currents of 180µA-4mA.  In some cases the current is very stable and high, like 2.3mA.  In other cases it will start fairly low (~20µA) and then ramp up to 180µA and continue to show instability.

That sounds a lot like a floaty pin. see if it's affected by a mild static field, e.g. wave a charged plastic comb near the PCB and see if it affects the current draw. If it does then it's almost certainly a floating pin

Thank you I will def try that.  I mentioned my VCC/2 testing and he said the pins were not floating.  We checked with my Fluke in ammeter mode to ground and looked for an increase in current?  But I have a number of people saying they are floating.  I will run your test tonight and report back.

Haha, yeah just a sales guy I claim no other skills.  Here's one of my baddies.  I have the + leg of supercap clipped so it's not part of circuit.

Here's a pic of PCB, let me know if you need any other details, I think schematic has already been posted.

Thx

Of course it's a black PCB    Green is better, it's easier to see the tracks on it and I might recommend cutting power traces but instead, you can remove parts.

I would try removing these parts from a bad board and see if it helps.  If it does, double check by putting them back on and retest.  Note the FW may need update to make it ignore the missing parts.

R14
U5
U2
U4

If that soldering is beyond you, try a local repair shop, even the little phone repair shops sometimes have people with good skills and equipment.

Haha, yeah but I like black and I picked the color...

I can easily remove those.  I presume you want this done on faulty board while monitoring current. 

R14=thermistor resistor
U5-Microchip EEPROM to store keys
U2-Max mic preamp, needed for firmware but maybe not?  Used at end of every hour
U4-3 axis accelerometer.  Not used in current firmware, so should be shut down.

I can unsolder all of this safely I have a rework station and Hakko iron.  Not sure I can get the acelerometer back on or even the preamp but I can try

Will report back tonight.

Thx

Black does look cool.  Maybe next time go with the after dark style, it's gorgeous and practical, you can see the copper on the outer layers.
https://blog.oshpark.com/2019/10/10/introducing-our-after-dark-black-fr-4-service/

Yes, use a bad board, remove 1 part at a time and test to see if that 'fixed' it.  If testing takes a long time, maybe remove 2 parts at a time or all the parts, whatever works for you.

The goal with this is to find where your extra current is going.  Once we figure that out then we add pull-ups/downs, improve the FW, etc.

For future reference or for an alternative to removing ICs:  what is handy for this is to run power for each IC through a resistor.  Ideally the PCB is designed with that but you could maybe cut tracks and insert resistors or with big parts like U5, you could lift pin 8 and add a resistor between pin 8 and VCC on the PCB. These resistors can be used to measure current to each IC and they can be removed to cut power to each IC, though you'd want to be careful not to send power to other pins while there is none to Vcc.  You can damage ICs doing that.

[tooki]

My desk is turning into a war zone here

I wanna see what the roll of stickers say on them! 😂

[sailah]

My desk is turning into a war zone here

I wanna see what the roll of stickers say on them! 😂

Haha, from my motorcycle racing days.  Free stickers if you figure out the problem !!

[Psi]

We did find a couple anomalies.  Pin #3 MAX9814_MICOUT is the output of the audio amplifier.  On a couple devices we were seeing 180-220mV on this pin with picoscope.  It did not correlate with high current but it should have been 0.

Be aware that piezo elements can have resistance as low as a few k ohms due to tiny cracks that can form during installation force.  Most of the time they are meg ohms/open circuit but not always.
I think it's caused by that fine metal power on the ceramic section to enables soldering. It can fall into any micro cracks and start to conduct from top to the metal plate.

It's common to put a capacitor in series to prevent this issue causing problems, which you don't have.

[sailah]

Last test for night, getting frustrated and been working on this all day

Took bad board, and removed:

R15-no change
U5-no change
U4-no change
U2-no change

Device is currently reading 2.38mA.  Should be 1.6µA.   So other than Murata module, all the ICs are removed.  Not sure I'm going to be able to mount them back on successfully.

[Psi]

You're not holding the PCB or probes while testing are you?
Skin contact will cause current to flow.

[sailah]

No I usually have it in a powder coated enclosure sitting on a mat.  Sometimes I'll hold the bare board but rare.  Probably power profiled 100s of these over the last few years and have many many extensive data logs to know almost immediately when the device is not working well.