Disabled power supply blows up expensive board... how?

[dmg]

We have just bought a new TTI CPX200D power supply for our lab, and it arrived today. We needed a power supply with remote sense capability to better power an RF board. The supply arrived this morning and we tested it for voltage and current and with some loads and everything looked OK, so this afternoon we proceeded to test the RF board. We had powered up the RF board previously with another supply without problems, but there were issues with voltage drop in the wiring.

We setup everything following both the supply manual and the RF board manual. The RF board gets its power from a connector, and outputs the two power sense lines, connected directly to the supply voltage and ground at the power connector. We twisted the sense wires together and connected them to the SENSE inputs of the supply, and connected the power wires to the output terminals. We switched the supply to remote sense, triple checked the connections polarity, adjusted voltage and current limit (the RF board is powered at 1.8V 1A, but the manufacturer recommends starting it with a 50mA current limit just in case and then raising it, so we set the current limit to 50mA) and proceeded to connect the power connector to the RF board, all of this with the supply output DISABLED.

Just when the power connector was starting to mate with its socket in the RF board, a tantalium capacitor connected across the power rail just at the RF board power connector blew up and started burning, and the board is now ruined. The RF board was worth 10000€. There was no other energy source for the RF board, no other active thing was connected to it. No instrument was connected either. Nothing. It has a single supply rail at 1.8V, we had succesfully powered it up before several times and, again, the output was DISABLED. With no other energy source available for the RF board, the energy responsible for blowing the capacitor (220 uF 16V tantalium cap) had to come from the power supply somehow. The thing is, we also had a multimeter connected accross the power lines monitoring the output voltage and it said 0V. Sense and power rails get tied together when the power connector is mated with the RF board, and these sense lines were the only non-monitored part we had. We checked afterwards that with the output DISABLED the sense lines remain at 0V and that there is no transient nor anything when power and sense lines get tied together.

Do you have any theory about how this happened?

EDIT:

Here's an update. During this time we tried simulating the board with a dummy load, consisting of an electrolytic capacitor with similar specs to those of the tantalum that blew up (electrolytics can tolerate much more without blowing up with toxic smoke) and found some interesting things.

If you proceed exactly as we proceeded when the incident happened (power supply channel turned off), nothing happens. The output stays at 0V, no transient, no nothing. No matter how you connect the supply to the dummy load, nothing happens. If the capacitor is pre-charged it just discharges with a nice exponential curve (so the output has confirmed bleeding resistors)

If you then turn on the supply with the dummy load already connected, the output just rises nicely to 1.8V, no overshoots or weird stuff.

BUT, if you turn on the power supply channel with the dummy load disconnected and you then hot-plug the dummy load, in some cases there's a several 10's of volts transient (>40V), almost 3 seconds long, and then the output stays at 1.8V. That scenario would have explained our issue... if it wasn't for the fact that the PSU supply channel was disabled. We triple-checked that it was indeed disabled before plugging anything. Both my colleague and I are 100% sure that it was off.

We'll keep investigating... (the manufacturer hasn't answered yet, we'll try new communication channels).

[Twoflower]

Have you measured the output of the PSU to the earth and/or to the GND of the RF board (in case it is connected to something else)?

I had a case a bad insulation made a bridge of the secondary to earth within my PSU and nearly killed my scope that way.

[dmg]

Have you measured the output of the PSU to the earth and/or to the GND of the RF board (in case it is connected to something else)?

I had a case a bad insulation made a bridge of the secondary to earth within my PSU and nearly killed my scope that way.

Yep, we tested supply negative and positive to ground isolation, It's properly isolated. No other connection to the RF board, just power. We're just puzzled.

EDIT: At least the cap was nice enough to spew the burning tantalium sparks sideways and not towards my face. For now we're assuming that if there was enough energy to blow a tantalium cap, there was enough energy to blow the main expensive IC at the board (no regulators in between... it's an evaluation board for a chip and it's made that way). We'll try to replace the cap and, if there's no hole in the PCB below it, try again and see if the IC is fried.

[Kleinstein]

A tantalum cap right at the input of a high current (e.g. 1 A) external supply is not such a good idea. Some tantalum caps don't like too high current spikes.  An expensive board with external supply should be protected against supply transients - for an RF board there might need to be RF filtering anyway - this could have saved the cap.

For a supply with external sense lines the sense lines , usually the sense wire should be twisted with the respective power wire and not the two sense wired together. Supplies have a very hard time to compensate external inductance and thus can get unstable worst case. So if the sense wires are twisted together, the power lines should be twisted as well - still not the preferred configuration.

Some voltage regulators might react a little odd with the sense lines not connected and thus possibly with transient voltages outside there operating rage. A good supply should be protected against this, but sometimes protection is not enough against ESD events on some inputs. For some strange reason a fast common mode change for the sense wires might get through, even with the output disabled.

It might be interesting to see, how high the input impedance is with the sense lines - if very high impedance, they might catch capacitive coupled line voltage and thus rather high amplitudes. So connecting sense and power part may not behave the same every time.

[dmg]

A tantalum cap right at the input of a high current (e.g. 1 A) external supply is not such a good idea. Some tantalum caps don't like too high current spikes.  An expensive board with external supply should be protected against supply transients - for an RF board there might need to be RF filtering anyway - this could have saved the cap.

For a supply with external sense lines the sense lines , usually the sense wire should be twisted with the respective power wire and not the two sense wired together. Supplies have a very hard time to compensate external inductance and thus can get unstable worst case. So if the sense wires are twisted together, the power lines should be twisted as well - still not the preferred configuration.

Some voltage regulators might react a little odd with the sense lines not connected and thus possibly with transient voltages outside there operating rage. A good supply should be protected against this, but sometimes protection is not enough against ESD events on some inputs. For some strange reason a fast common mode change for the sense wires might get through, even with the output disabled.

It might be interesting to see, how high the input impedance is with the sense lines - if very high impedance, they might catch capacitive coupled line voltage and thus rather high amplitudes. So connecting sense and power part may not behave the same every time.

The design is not ours, it's the official evaluation board from the manufacturer of a pre-release RF IC. They recommend the power supply with sense lines stuff. If it was me who designed the thing, I'd provide the power locally through a regulator but...

You're right about the twisting, we did it as you say, I explained it wrong before. Tantalium smoke inhaling... you know.

But apart from all of this, the supply was disabled. I mean, output OFF. Nothing should blow with output OFF.

[duak]

I wonder if when the supply output is disabled it is truly disconnected or is it connected but only set to deliver 0 V?  In the latter case, could quickly connecting a capacitor to the sense leads trick the supply into generating an output pulse?

Best o' luck!

[jmelson]

Something we always do on remote-sense supplies is put some resistors between +out and +sense, and -OUT and -sense.  This prevents the supply from running away if the sense circuit is not actually connected.  I usually use about 100 Ohms for these, but it depends on the load the remote sense input imposes.

Not sure if your supply has these, you could just use a DVM to see if they have such in their circuit.  If NOT, then I'd definitely add this.

Jon

[jmelson]

Something we always do on remote-sense supplies is put some resistors between +out and +sense, and -OUT and -sense.  This prevents the supply from running away if the sense circuit is not actually connected.  I usually use about 100 Ohms for these, but it depends on the load the remote sense input imposes.

Not sure if your supply has these, you could just use a DVM to see if they have such in their circuit.  If NOT, then I'd definitely add this.

Jon

[dmills]

I had a lovely gotcha like that, lab power supply turned out to have a honking great cap across the output.

Had been doing something with 12V or so, switched off, reset the output voltage to 2.5V or so, plugged in the load....

Swearing ensued.

Regards, Dan.

[julianhigginson]

Owch!

I'd be tempted to try something.... Get a cap the same as on your board (need a new one for the board, anyway) and a 2r resistor in parallel. And maybe a connector, to make a passive version of your board's load...

Put an oscilloscope across it, set to trigger on any voltage rise above 1v..

Do your thing with the PSU, then plug it in.

See if you can see anything at all that might kill your board.

[dmg]

Something we always do on remote-sense supplies is put some resistors between +out and +sense, and -OUT and -sense.  This prevents the supply from running away if the sense circuit is not actually connected.  I usually use about 100 Ohms for these, but it depends on the load the remote sense input imposes.

Not sure if your supply has these, you could just use a DVM to see if they have such in their circuit.  If NOT, then I'd definitely add this.

Jon

This particular supply has a limited 0.5V compensation range for the SENSE lines

I had a lovely gotcha like that, lab power supply turned out to have a honking great cap across the output.

Had been doing something with 12V or so, switched off, reset the output voltage to 2.5V or so, plugged in the load....

Swearing ensued.

Regards, Dan.

That's strange, even my personal 100€ PSU has bleeding resistors across the output caps to avoid that...  In this case, I was explaining my colleague how the supply works just before that, and the last thing I did was toggling the PSU while set to 1.8V, then I locked the dials and, with the supply switched off, I proceeded to connect power, and boom.

Owch!

I'd be tempted to try something.... Get a cap the same as on your board (need a new one for the board, anyway) and a 2r resistor in parallel. And maybe a connector, to make a passive version of your board's load...

Put an oscilloscope across it, set to trigger on any voltage rise above 1v..

Do your thing with the PSU, then plug it in.

See if you can see anything at all that might kill your board.

I'll definitelly do that, but inside a fume hood and with safety goggles. I'm lucky to have only experienced burnt tantalium scent about 5 times in my life and sincerely I don't really want to get that privilege again. I love when the damn thing gets exposed to air after blowing off and initiates a self-sustaining fire even if you cutoff the power, and you can only enjoy while it drills a hole through your PCB. Nasty stuff.. I try to avoid them like the plague in my designs in favour of ceramics/polymer, but sometimes they're the best choice....

[Mr. Scram]

I had a lovely gotcha like that, lab power supply turned out to have a honking great cap across the output.

Had been doing something with 12V or so, switched off, reset the output voltage to 2.5V or so, plugged in the load....

Swearing ensued.

Regards, Dan.

I have to test that one with my own supplies, to avoid a surprise.

[xavier60]

Keeping in mind that the output was disabled when the damage occurred, I'm curious about what this means, "Sense and power rails get tied together when the power connector is mated with the RF board,"?

[SeanB]

I would guess the power supply disables the output by setting voltage to zero. The disconnected sense leads allowed the terminals to float from leakage to some voltage, as there was no feedback from any sensing to downprogram the voltage.

[Bud]

Something we always do on remote-sense supplies is put some resistors between +out and +sense, and -OUT and -sense.

Do you connect the resistors right at the PS output terminals ?

[dmg]

I would guess the power supply disables the output by setting voltage to zero. The disconnected sense leads allowed the terminals to float from leakage to some voltage, as there was no feedback from any sensing to downprogram the voltage.

But according to the manual, the correction range through sense is 0.5V, that should mean the output should have stayed to within +-0.5V during this event, and that's not enough to blow a tantalium cap.

I mean, I don't expect to have to take these sort of precautions when connecting a disabled 900€ PSU to something. In fact, how else could I protect the circuit against startup transients?

Sure thing, the RF board is poorly designed (no local decoupling near the chip, too much imput capacitance after too long wires (they provide the power wiring with the kit), no local regulator, having to use a supply with sense capability to compensate for voltage drop during transients as there's no internal regulator, no TVS, no anti-parallel diode at the input...) but still... a disabled PSU should be disabled.

Keeping in mind that the output was disabled when the damage occurred, I'm curious about what this means, "Sense and power rails get tied together when the power connector is mated with the RF board,"?

That means that the power wire harness  has 4 independent wires, and the mating on-board connector has two pairs of pins connected together, one to the supply rail, the other to ground. One pin of each pair is meant to supply voltage, the other to sense the voltage right at the load.

Let me take a wild guess: RFSoC?

We work a lot with xilinx parts and have quite a lot of Xilinx eval boards from Virtex 2 Pro era to modern Zynq stuff. Haven't tested RFSoC's yet, but I'm pretty sure that Xilinx would never design an eval board with such design flaws in the supply section. Moreso when their devices are notorious for their complicated power sequencing scheme, that I myself have suffered already.
It's a way more obscure IC for a niche application, can't disclose sorry.

[abraxa]

Since this is a brand-new PSU, did you contact the manufacturer for comment?

[Maxlor]

Could it be a fault in the PSU that lead to AC voltage on the output? You'd probably have missed that when you checked with the DMM.

[jpb]

I am no expert, but one random thought I've had is perhaps the tantalum was charged when you connected it and the power supply suddenly shorted it causing a large discharge current? (It perhaps had been charged when the board was previously tested.)

I've no idea if this is a possibility but tantulums don't like large currents and I guess this also includes a discharge current.

[dmg]

Since this is a brand-new PSU, did you contact the manufacturer for comment?

Yes, waiting for their answer!

Could it be a fault in the PSU that lead to AC voltage on the output? You'd probably have missed that when you checked with the DMM.

Didn't check that, but I'm pretty sure I touched the output leads when I was testing it and didn't get a shock. Will test anyway, but if that's the fault then well...

I am no expert, but one random thought I've had is perhaps the tantalum was charged when you connected it and the power supply suddenly shorted it causing a large discharge current? (It perhaps had been charged when the board was previously tested.)

I've no idea if this is a possibility but tantulums don't like large currents and I guess this also includes a discharge current.

Not likely, the board had been sitting inside its box for almost a week, any possible charge should have leaked already. Also, we checked today and the main IC on board is totally fried, so there was definitelly some spike or something.

The good news is that the tantalum didn't burn a hole through the board, the PCB is undamaged. We'll try to swap the main IC and repair the board.

[xavier60]

However it happened, rather than high voltage, maybe it was reverse polarity. This would more likely explain the tantalum capacitor failing.

[xavier60]

Just found in the power supply specs  "Reverse protection by diode clamp for currents to 3A". So reverse polarity is unlikely.

[SilverSolder]

Imagine this:   The tantalum was charged when you connect the PSU.   The PSU voltage servo circuit suddenly sees this voltage on its sense wires.  It concludes that its output is wrong, and immediately attempts to correct the situation by applying the opposite voltage to bring things under control.   Unfortunately it overshoots the zero volt point and ends up in negative territory (even for a short instant), which blows the tantalum capacitor (which is sensitive to reverse voltages).

You can test this by connecting a charged capacitor of about the same size to the supply and watch the party on a scope?

[dom0]

The PSU can't do that (apply negative voltage), it's not a 4Q unit.

[dmg]

Here's an update. During this time we tried simulating the board with a dummy load, consisting of an electrolytic capacitor with similar specs to those of the tantalum that blew up (electrolytics can tolerate much more without blowing up with toxic smoke) and found some interesting things.

If you proceed exactly as we proceeded when the incident happened (power supply channel turned off), nothing happens. The output stays at 0V, no transient, no nothing. No matter how you connect the supply to the dummy load, nothing happens. If the capacitor is pre-charged it just discharges with a nice exponential curve (so the output has confirmed bleeding resistors)

If you then turn on the supply with the dummy load already connected, the output just rises nicely to 1.8V, no overshoots or weird stuff.

BUT, if you turn on the power supply channel with the dummy load disconnected and you then hot-plug the dummy load, in some cases there's a several 10's of volts transient (>40V), almost 3 seconds long, and then the output stays at 1.8V. That scenario would have explained our issue... if it wasn't for the fact that the PSU supply channel was disabled. We triple-checked that it was indeed disabled before plugging anything. Both my colleague and I are 100% sure that it was off.

We'll keep investigating... (the manufacturer hasn't answered yet, we'll try new communication channels).