Sharing a event last week that reminded me to not be lazy and
always check other people's equipment.
A few weeks ago we received an opportunity to make a modified version of an existing product for a potential new customer. It's a version we'd thought about for a couple of years but didn't have a ready-made customer for it, so with being busy doing lots of other things we never bothered to do it. However, that meant we had lots of the basic details already worked out so hand-building some breadboards for R&D, and then some prototypes for demonstration at the customer, went pretty quickly. We were on the plane last Sunday and in their building Monday morning.
This is a marine application - devices meant for installation on watercraft. As such it's an "automotive" environment, 12VDC lead acid cells backed by one or more alternators on an engine, and a CAN network at 250KHz. This is stuff we know very well with many years of experience, multiple products shipping, and many thousands of products in the field.
Since they're in the same environment, they have a nice test bench set up with everything our device should connect to. Power comes from a big benchtop analog power supply. At first look I was jealous of this beast... sometimes the loads we control have inrush currents in the many tens of amps and all of our supplies crowbar, and the array of lead acid cells we keep around for such tests are annoying to use and maintain. This box was beautiful in its simplicity: A power switch and two terminals on the front, a power cord and fuse holder in the back. That's it. No adjustments, no selectors. The white silkscreen on the front could not be misunderstood: Just the manufacturer's name and "13.8 VDC 50A" under the terminals. I could imagine the boat anchor power transformer lurking inside. This was a supply tailor-made for handling large automotive loads.
We only needed two of our devices to demonstrate, but we had hand-built five "just in case". Each one took many hours of careful assembly, using our standard version with a daughterboard of perfboard glued to the PCB.
We hooked up the first two and turned on the power supply. We enabled the first device (they operate alternately, in pairs) and everything worked briefly. Then there was a very audible
POP. I heard it, my companion heard it, the customer's staff heard it. Everything stopped working. There was no covering it up.
I was thunderstruck. Baffled. Absolutely mystified. I had personally tested all five of these devices, put them through rigorous abuse. We were spending thousands of dollars on a trip to the customer, and I wanted to be darned certain the demo went perfectly. AND that we had backups in case anything went wrong. This is not my first rodeo. I like to be prepared and look good in front of the customer.
Our device was dead. Reapplying power did nothing. It didn't smoke, but I may have been able to smell something (though that could have been residual odors from the assembly process).
I did a bunch of hand-waving and distracting, as did my companion, while I swapped in a replacement device.
Click.
POP. Same result.
WTF?!?OK, maybe possibly perhaps one of the devices got "damaged" on the trip (though I personally carried all of them in individual antistatic bags in my padded backpack which was in my possession the entire time). But the odds were zero of two getting "damaged" in the same way, so as to fail in exactly the same manner. And now we're down to "just" three devices. We can "afford" to test, and lose, only one more before the trip is a failure. And our reputation is falling in real time.
On trips like these I tend to bring a mini-development environment so I'm prepared for most things. I didn't bring the portable scope, but I did bring a Fluke DMM. On a whim, running out of excuses and distracting magic tricks and starting to sweat a little, I whipped out the DMM
and measured the voltage coming out of their supply.
29.1 VDC. Keep in mind this was a simple analog supply. Not a battery charger. Not a jump starter. It had typical threaded banana jacks on its front panel. It was meant for a bench environment. And it was labeled 13.8VDC right on the front panel, with no adjustments anywhere on its six outside surfaces.
Hey, we know how to design for nasty automotive electrical systems. We spend extra money and PCB real estate on power protection components and aggressively test our designs. We expect reversed power connections. We expect spikes of both polarities, spikes amplitudes in the hundreds of volts, alternator field collapse, etc. But we do NOT expect sustained DC voltage well over 2X nominal backed by 50A of sustained current capacity.
Having identified the culprit, I whipped out the little switcher that I built a while back to do hotel-based development. It confirmed that our remaining three devices worked fine, but that little supply didn't have the grunt to power the loads in question. At this point I knew for certain we could give a great demo and save the day - we just needed a reliable source of 13.8VDC.
When scheduling the trip I had asked the customer if they had lead acid cells and they had assured me they had a full array of power sources. On the spot, it turned out this 29V power supply was pretty much it. I asked again if they had batteries - nope.
Desperate, realized we had brought a solution with us. There was going to be a demo today one way or another. They had a full machine shop, so I asked for a wrench so I could go remove the battery from our rental car. While they were finding that, the general manager said "Hey, we have forklifts...."
And so they drove one of their propane-fueled forklifts over to the test bench. We opened the engine compartment and while my companion handled our devices, I held two wires against the battery terminals of the forklift. It all worked perfectly, just as we knew it would. We demonstrated multiple modes of operation, different configurations, everything.
They were very impressed. They apologized for their supply. And we came away with a great new working relationship that is already yielding benefits to both parties.
Back home, I personally did the post-mortem on the two blown devices. I believe the sequence of events was as follows:
1) TVS's have this "spread" of voltage thresholds and specifying the best part for the job is a study in compromises. Too low a voltage and the TVS is conducting too often; too high a voltage and the downstream components aren't protected. The unidirectional TVS we have settled on has an initial threshold of 16V and a clamping voltage of 26V, and has been working great in thousands of devices in the field for many years so we have a lot of confidence in its behavior in a normal (awful) automotive electrical environment. But in THIS environment, the TVS saw a sustained voltage of 29V - well above its 26V clamping voltage where the TVS turns fully on in an attempt to sink enough current to bring the voltage down. "Sink enough current" in this context is
50 amperes. The max forward voltage of the TVS is spec'd at ~3V, so the TVS was trying to conduct a sustained ~50A and dissipate a sustained ~150W. That there was a delay of any length between power on and the POP is a miracle.
2) With ~50A flowing, the forward biased polarity protection diode (rated for 1000V 1A) failed short. This didn't really affect the outcome, just added insult to injury by raising the voltage experienced by the downstream TVS by whatever the diode drop had been previously.
3) Next the TVS couldn't take it anymore and failed short. This is the normal mode of failure for a TVS. At this point we had the full voltage and full current of the supply running through the connector, the diode, and TVS, and the PCB traces that connect all of them.
4) Finally, one of the PCB traces acted as a fuse and blew, opening the circuit and ending the nightmare. The same trace blew on both devices so we know which of the involved traces is the weakest!
Moral of the story: Just because a nice looking fixed voltage bench power supply reads "13.8V" on its front panel does not mean you can trust it. You might think you can... it has no adjustments so it cannot be misadjusted... a motor connected to it moments before worked and didn't fry, giving you a false sense of confidence... but don't be fooled. Bring a meter that you trust and confirm. It might save you a trip.