EEVblog 1511 - My Solar Analytics System FAILED

[EEVblog]

The magic smoke was released from my solar analytics monitoring system. What was the culprit?
Also Dumb Dave™ gets ZAPPED!

[floobydust]

It's good it didn't burn up the universe 
Normally I would suspect an overload on the secondary, a shorted supercap etc. was what killed the rectifier diode.
But the second failure is that resistor. Transformer Würth 7508110151 is for a 5V 1A output. You could trace the damaged resistor to it, it looks like part of a snubber.
I dislike designs where components end up being the fuse. That's not good. I'll bet the panel board feed to the PSU has no sane value for a fuse. I add that to all voltage sense devices because the amperage available is quite high if there is carbon tracking on the primary side and fusible resistors are not really good fuses at all. Bad component choice there.
Why is there brown varnish or flux all over the place, it's disgusting.

P.S. Did the multimeter probe draw a small arc at 2:18-2:22? It looked strange, I've only seen that with HF HV.

[EEVblog]

P.S. Did the multimeter probe draw a small arc at 2:18-2:22? It looked strange, I've only seen that with HF HV.

looks lik it could just be reflected light off the metal screw.

[EEVblog]

Normally I would suspect an overload on the secondary, a shorted supercap etc. was what killed the rectifier diode.

Both caps seems to charge up with the ohmeter.

[wraper]

Does not look that bad. Charring looks more like magic smoke deposited on the PCB. Sure there is some PCB discoloration but I don't think PCB material carbonized and became conductive. I would first try to clean it and then see if it's worth repairing. Of course even if PCB became conductive, bad material can be milled out.

[Ranayna]

A YT comment mentioned it, and i would tend to agree: The supercaps don't look healthy. It really looks like they leaked, and that might also be the stuff that contaminated the cardedge connector.

A bit offtopic: I find it very fascinating that not only your fusebox is outside the house, but the central heating burner as well. Shows how different the climate in Australia is

[EEVblog]

A bit offtopic: I find it very fascinating that not only your fusebox is outside the house, but the central heating burner as well. Shows how different the climate in Australia is

No such thing as a "central heater burner" here. That is my gas hot water system.
To heat the house in winter (if needed) everyone use reverse cycle airconditioners.

[station240]

I'd fix it, assuming Solar Analytics don't ask for it back.
Reason being, a longer term power meter would be handy for the lab, could just power from an internal 5V source (see below).
Would need to go in the Ultrasonic cleaner first.

That broken in half resistor has the marking 151 (150 ohm).
Datasheet for the 5V 1A out Flyback transformer https://www.we-online.com/catalog/datasheet/7508110151.pdf Found it in the TI reference design section.

The two supercaps appear to be in series, which makes sense as they are 2.7V rated. Each has a resistor across it, (R14 & R16), but R14 is also missing.
I have to wonder if that missing resistor lead to one of the supercaps getting an uneven share of the voltage.

Dave, any change of a reverse engineering of the secondary side ?
I've not seen a PSU with supercaps on the secondary, and it looks like there are extra components for a precharge circuit.

[thm_w]

A YT comment mentioned it, and i would tend to agree: The supercaps don't look healthy. It really looks like they leaked, and that might also be the stuff that contaminated the cardedge connector.

A bit offtopic: I find it very fascinating that not only your fusebox is outside the house, but the central heating burner as well. Shows how different the climate in Australia is

Datasheet for the cap: https://media.digikey.com/pdf/Data%20Sheets/Nesscap%20URL%20links/ESHSR-0005C0-002R7.pdf
Rated for 1,500hrs at 65C which is nothing.

Also, they derate the voltage depending on the temperature, it drops to a 2.3V rating at 85C.
So question is what rail was it running on, 3.3V or 5V?

[floobydust]

It looks like two related component failures and lucky the parts literally fell off the PC board as the safety catch.
If it was installed in a different orientation and the diode stayed put, it could have had a bad ending.
I don't see any product regulatory or safety approvals and with stuff like that, good engineering intentions... but without formal testing I feel that is high risk to run in a panel. Does it take Cat. III, how's that fuse when things short out? I think the fusible resistor? is a mistake. They have very high fuse ratings 30-60x rated power, and have no interrupt ratings. EE's have to do their homework with them and not be naive.

I think the power supply is undersized for the cell modem's max. TX current, so the strategy is using super caps to help. They might also provide a brief battery action for graceful network shutdown at a mains failure, as I found out the hard way you make orphaned IP sockets when you yank power to a cell modem on the network. Cinterion EHS6 peak TX worst case GSM 1.3-1.8A and I don't see any antenna connected which would give bad SWR. Supercap C4 sits high up and does not look good at the bottom, I would take it out.

I imagine there's a danger running things with the CT's unplugged. I worked with big 800/5A CT's and company practice was to always short them, never run them open with the burden resistor disconnected.

[SeanB]

New one will fit in the same box, just will have a nice big DIN rail box to hold all the breakers and other DIN rail mounted units, with the meters now all being on the top row in a line. Plenty of space then for extension.

When you replace the heater, insulate all the hot water pipes both outside and inside the roof, so as to minimise the heat loss from the long runs.

The meter likely the one supercap went very leaky, and drained itself, leaving the other to handle 5V on it's own, leading to it spewing electrolyte on the PCB, filling the interconnect with the goop. Then the constant high current finally desoldered the diode, and then the PSU was running open output, leading to the snubber failing, and then the little dinky SMPS chip got killed when the tiny little mosfet blew it's bondwire off. Thus no discharge resistor, dual purpose discharge and startup resistor, with the tiny ceramic capacitor being grossly overvolted, as they used the same diode as secondary side power supply and snubber, one less reel to have on the PNP.

[martinwag]

had the same issue a few years back, see https://www.eevblog.com/forum/projects/supercap-leaking-acid/msg992195/#msg992195

same symptoms, same cap brand / series...
- leaks acid -> shorts, defective components, defective vias, broken traces.
- keeps rated capacitance
If the pcb has damage throw away the device, it can't be repaired reliable.

Nesscap was bought by maxwell a few yeas back, the caps now come in blue heatshrink.

[thm_w]

had the same issue a few years back, see https://www.eevblog.com/forum/projects/supercap-leaking-acid/msg992195/#msg992195

same symptoms, same cap brand / series...
- leaks acid -> shorts, defective components, defective vias, broken traces.
- keeps rated capacitance
If the pcb has damage throw away the device, it can't be repaired reliable.

Nesscap was bought by maxwell a few yeas back, the caps now come in blue heatshrink.

So you were running them at 2.5V, probably much better condition than Daves unit was in, and they still failed.

[tszaboo]

I'd fix it, assuming Solar Analytics don't ask for it back.
Reason being, a longer term power meter would be handy for the lab, could just power from an internal 5V source (see below).
Would need to go in the Ultrasonic cleaner first.

That broken in half resistor has the marking 151 (150 ohm).
Datasheet for the 5V 1A out Flyback transformer https://www.we-online.com/catalog/datasheet/7508110151.pdf Found it in the TI reference design section.

The two supercaps appear to be in series, which makes sense as they are 2.7V rated. Each has a resistor across it, (R14 & R16), but R14 is also missing.
I have to wonder if that missing resistor lead to one of the supercaps getting an uneven share of the voltage.

Dave, any change of a reverse engineering of the secondary side ?
I've not seen a PSU with supercaps on the secondary, and it looks like there are extra components for a precharge circuit.

These modems will eat about 10W peaks at 3.3-3.6V if they have 2G GSM in them. Looks like the engineer wanted to save a dollar, and put these in to reduce those peaks. I don't think it's a particularly good design decision to do, running 2.7V rated caps at 2.5V without active balancing, without controlled charging circuit, and with quite high (though OK by the datasheet) ripple currents.

A potted 10W rated, CE compliant, and EMC tested power supply is about 6-7 EUR, and it doesn't spill it's guts.
Also, none of the large components had silastic on them. Not good.

[EEVblog]

These modems will eat about 10W peaks at 3.3-3.6V if they have 2G GSM in them. Looks like the engineer wanted to save a dollar, and put these in to reduce those peaks. I don't think it's a particularly good design decision to do, running 2.7V rated caps at 2.5V without active balancing, without controlled charging circuit, and with quite high (though OK by the datasheet) ripple currents.

The designer has confirmed that pulse peaks was an advnatage back in old 2G designs. This is 3G. And the latest design they are sendign me (presumably 4G?) only has a single supercap.

Update form the video comments I posted:

UPDATE: quite a few eagle eyed viewers spotted that R14 was missing. R14 is one of the balance resistors across the supercaps. That falling off could certainly have eventually triggered the event we saw here.
I cannot find the resistor, but upon inspection of the pads it was clear that the resistor was soldered in there at one point, but the joint appears like it cracked so to speak. So it could have been a poor solder joint that ultimately lead to the failure. Indeed, the unit likely wouldn't have survived long at all if R14 was missing from production. The resistor could have fallen out on handling and transport to the lab.
I have spoken to the designer and the balance resistors were chosen to prevent imbalance and hence overvoltage in the supercaps, and there have been no other reported failures in over 70k units installed. He also concluded that the unit wouldn't have survivded this long if missing from production. In any case the latest design they are sending does not have series supercaps like this. I'll try and do a teardown when I get it.

[EEVblog]

Also, they derate the voltage depending on the temperature, it drops to a 2.3V rating at 85C.
So question is what rail was it running on, 3.3V or 5V?

2.3V each confirmed by the designer.

[Ed.Kloonk]

Three cheers for the designer.

Try getting that information and frankness for any other similar product produced in other lands.

[tszaboo]

These modems will eat about 10W peaks at 3.3-3.6V if they have 2G GSM in them. Looks like the engineer wanted to save a dollar, and put these in to reduce those peaks. I don't think it's a particularly good design decision to do, running 2.7V rated caps at 2.5V without active balancing, without controlled charging circuit, and with quite high (though OK by the datasheet) ripple currents.

The designer has confirmed that pulse peaks was an advnatage back in old 2G designs. This is 3G. And the latest design they are sendign me (presumably 4G?) only has a single supercap.

They can probably get away with 3G only designs there. Here in EU, all the countries are randomly choosing what network to keep on operating. Luckily LTE CAT-M seems to be universally supported, and modems have less than 500mA peak currents. The difference is night and day design wise, the new tech is just so much better. NB-IOT would be even better with it's 200mA peaks, if it wasn't for the lack of compatibility, issues with roaming, and FW guys reliance on OTA updates to make a product work. For a fixed installation it really is a game changer.

[HackedFridgeMagnet]

Enjoyed the fault finding and teardown.
fyi
I just wanted to point out the earth "block" is the green and yellow mess of wires in the photo. Some of the earth wires are probably soldered together under the tape.
My house used to be similar. I am sure glad they dont do this anymore.

There were two neutral blocks , one incoming with the blue wire and the green/yellow men connection.
The other neutral block was a post rcd neutral so that circuits can share a single rcd.

FWIW I was also suprised at the creepage distances on the module. Seemed a bit small for something within a switchboard. But i dont have details to back up this opinion.

[eti]

It was worth watching to watch you getting zapped! 😂😆

Obviously the unauthorised repair agent deterrent circuit was fully working 😁