Never Trust Other People's Equipment!

[IDEngineer]

Sharing a event last week that reminded me to not be lazy and always check other people's equipment.

A few weeks ago we received an opportunity to make a modified version of an existing product for a potential new customer. It's a version we'd thought about for a couple of years but didn't have a ready-made customer for it, so with being busy doing lots of other things we never bothered to do it. However, that meant we had lots of the basic details already worked out so hand-building some breadboards for R&D, and then some prototypes for demonstration at the customer, went pretty quickly. We were on the plane last Sunday and in their building Monday morning.

This is a marine application - devices meant for installation on watercraft. As such it's an "automotive" environment, 12VDC lead acid cells backed by one or more alternators on an engine, and a CAN network at 250KHz. This is stuff we know very well with many years of experience, multiple products shipping, and many thousands of products in the field.

Since they're in the same environment, they have a nice test bench set up with everything our device should connect to. Power comes from a big benchtop analog power supply. At first look I was jealous of this beast... sometimes the loads we control have inrush currents in the many tens of amps and all of our supplies crowbar, and the array of lead acid cells we keep around for such tests are annoying to use and maintain. This box was beautiful in its simplicity: A power switch and two terminals on the front, a power cord and fuse holder in the back. That's it. No adjustments, no selectors. The white silkscreen on the front could not be misunderstood: Just the manufacturer's name and "13.8 VDC 50A" under the terminals. I could imagine the boat anchor power transformer lurking inside. This was a supply tailor-made for handling large automotive loads.

We only needed two of our devices to demonstrate, but we had hand-built five "just in case". Each one took many hours of careful assembly, using our standard version with a daughterboard of perfboard glued to the PCB.

We hooked up the first two and turned on the power supply. We enabled the first device (they operate alternately, in pairs) and everything worked briefly. Then there was a very audible POP. I heard it, my companion heard it, the customer's staff heard it. Everything stopped working. There was no covering it up.

I was thunderstruck. Baffled. Absolutely mystified. I had personally tested all five of these devices, put them through rigorous abuse. We were spending thousands of dollars on a trip to the customer, and I wanted to be darned certain the demo went perfectly. AND that we had backups in case anything went wrong. This is not my first rodeo. I like to be prepared and look good in front of the customer.

Our device was dead. Reapplying power did nothing. It didn't smoke, but I may have been able to smell something (though that could have been residual odors from the assembly process).

I did a bunch of hand-waving and distracting, as did my companion, while I swapped in a replacement device.

Click. POP. Same result. WTF?!?

OK, maybe possibly perhaps one of the devices got "damaged" on the trip (though I personally carried all of them in individual antistatic bags in my padded backpack which was in my possession the entire time). But the odds were zero of two getting "damaged" in the same way, so as to fail in exactly the same manner. And now we're down to "just" three devices. We can "afford" to test, and lose, only one more before the trip is a failure. And our reputation is falling in real time.

On trips like these I tend to bring a mini-development environment so I'm prepared for most things. I didn't bring the portable scope, but I did bring a Fluke DMM. On a whim, running out of excuses and distracting magic tricks and starting to sweat a little, I whipped out the DMM    and measured the voltage coming out of their supply.

29.1 VDC.

      

Keep in mind this was a simple analog supply. Not a battery charger. Not a jump starter. It had typical threaded banana jacks on its front panel. It was meant for a bench environment. And it was labeled 13.8VDC right on the front panel, with no adjustments anywhere on its six outside surfaces.

Hey, we know how to design for nasty automotive electrical systems. We spend extra money and PCB real estate on power protection components and aggressively test our designs. We expect reversed power connections. We expect spikes of both polarities, spikes amplitudes in the hundreds of volts, alternator field collapse, etc. But we do NOT expect sustained DC voltage well over 2X nominal backed by 50A of sustained current capacity.

Having identified the culprit, I whipped out the little switcher that I built a while back to do hotel-based development. It confirmed that our remaining three devices worked fine, but that little supply didn't have the grunt to power the loads in question. At this point I knew for certain we could give a great demo and save the day - we just needed a reliable source of 13.8VDC.

When scheduling the trip I had asked the customer if they had lead acid cells and they had assured me they had a full array of power sources. On the spot, it turned out this 29V power supply was pretty much it. I asked again if they had batteries - nope.

Desperate, realized we had brought a solution with us. There was going to be a demo today one way or another. They had a full machine shop, so I asked for a wrench so I could go remove the battery from our rental car. While they were finding that, the general manager said "Hey, we have forklifts...."

And so they drove one of their propane-fueled forklifts over to the test bench. We opened the engine compartment and while my companion handled our devices, I held two wires against the battery terminals of the forklift. It all worked perfectly, just as we knew it would. We demonstrated multiple modes of operation, different configurations, everything.

They were very impressed. They apologized for their supply. And we came away with a great new working relationship that is already yielding benefits to both parties.

Back home, I personally did the post-mortem on the two blown devices. I believe the sequence of events was as follows:

1) TVS's have this "spread" of voltage thresholds and specifying the best part for the job is a study in compromises. Too low a voltage and the TVS is conducting too often; too high a voltage and the downstream components aren't protected. The unidirectional TVS we have settled on has an initial threshold of 16V and a clamping voltage of 26V, and has been working great in thousands of devices in the field for many years so we have a lot of confidence in its behavior in a normal (awful) automotive electrical environment. But in THIS environment, the TVS saw a sustained voltage of 29V - well above its 26V clamping voltage where the TVS turns fully on in an attempt to sink enough current to bring the voltage down. "Sink enough current" in this context is 50 amperes. The max forward voltage of the TVS is spec'd at ~3V, so the TVS was trying to conduct a sustained ~50A and dissipate a sustained ~150W. That there was a delay of any length between power on and the POP is a miracle.

2) With ~50A flowing, the forward biased polarity protection diode (rated for 1000V 1A) failed short. This didn't really affect the outcome, just added insult to injury by raising the voltage experienced by the downstream TVS by whatever the diode drop had been previously.

3) Next the TVS couldn't take it anymore and failed short. This is the normal mode of failure for a TVS. At this point we had the full voltage and full current of the supply running through the connector, the diode, and TVS, and the PCB traces that connect all of them.

4) Finally, one of the PCB traces acted as a fuse and blew, opening the circuit and ending the nightmare. The same trace blew on both devices so we know which of the involved traces is the weakest!

Moral of the story: Just because a nice looking fixed voltage bench power supply reads "13.8V" on its front panel does not mean you can trust it. You might think you can... it has no adjustments so it cannot be misadjusted... a motor connected to it moments before worked and didn't fry, giving you a false sense of confidence... but don't be fooled. Bring a meter that you trust and confirm. It might save you a trip.

[genghisnico13]

I didn't read anything about a PolySwitch, in that case (with a high enough voltage rating) should protect the device.

[IDEngineer]

I didn't read anything about a PolySwitch, in that case (with a high enough voltage rating) should protect the device.

Perhaps. But polyswitches say "speed of response for a PTC is similar to the time delay of a slo-blo fuse", so in the environment in question the TVS would still have to sustain ~50A and ~150W for, what, several hundred milliseconds? TVS's can be damaged and still be semi-functional so this raises the threat of having a marginal device rather than one which fails hard, immediately revealing that there is a problem.

You have a great point but polyswitches aren't a panacea. I'm take another look at their specs but I'm not sure they're a good fit. And besides, I haven't yet seen the 12V automotive battery that accidentally generates twice the expected voltage from its chemistry.    We're not in a rush to redesign for this condition as we consider it highly unlikely to occur in an actual watercraft environment.

[radiolistener]

Here is protection circuit which allows to protect equipment from issues like overvoltage and reverse polarity for high current.

It should help in your case.

[Mr. Scram]

Don’t trust your own, for that matter. And definitely don’t trust yourself.

[IDEngineer]

Here is protection circuit which allows to protect equipment from issues like overvoltage and reverse polarity for high current.

Yep, if you have more room you can implement more elaborate solutions. The PCB in question is 2x2 inches and nothing but the connector (which consumes 25% of that real estate) can exceed 0.25in in height.

As I said, with many thousands of devices in the field, our existing solution is proven in the nasty environment of an automotive electrical system. It also takes up much less room than (for example) a relay. I was sharing the experience in case I'm not the only one who forgets to check other people's equipment. {grin}

[thinkfat]

If you want to use a TVS diode for overvoltage protection, you must combine it with a fast acting fuse. A polyfuse is waaay too slow. A crowbar circuit with an SCR would work better with a polyfuse (tried and tested). The T in TVS gives away what you would use it for.

[IDEngineer]

I prefer the term "tranzorb" though most people think "TVS" so that's what I typed.

Our tranzorb's turnon time is, according to the spec sheet, 1ps typical (yep, picosecond). No (poly)fuse is going to come within many orders of magnitude. Remember, we aren't expecting the tranzorb to deal with steady-state overvoltage conditions. Its job is to absorb transients, hence the name.

Since this is the second comment involving a fuse: This is a potted module which gets certified to SAE J1171 for ignition-proof operation in engine compartments. All components are embedded in epoxy. Replacement of anything, including a fuse, is impossible. So far we've had zero returns from the field after several years and thousands of installed units. We're comfortable continuing production without adding a fuse.

[nctnico]

I didn't read anything about a PolySwitch, in that case (with a high enough voltage rating) should protect the device.

Polyswitches are not the best option. In this case I'd use a fuse in the design (internal or external). That would have prevented the device from being blown up. Since it appearantly is automotive I'd expect the device to be behind a fuse anyway so why no add that to a demo setup just in case?

[thinkfat]

I prefer the term "tranzorb" though most people think "TVS" so that's what I typed.

Our tranzorb's turnon time is, according to the spec sheet, 1ps typical (yep, picosecond). No (poly)fuse is going to come within many orders of magnitude. Remember, we aren't expecting the tranzorb to deal with steady-state overvoltage conditions. Its job is to absorb transients, hence the name.

Since this is the second comment involving a fuse: This is a potted module which gets certified to SAE J1171 for ignition-proof operation in engine compartments. All components are embedded in epoxy. Replacement of anything, including a fuse, is impossible. So far we've had zero returns from the field after several years and thousands of installed units. We're comfortable continuing production without adding a fuse.

Well no problem with that of course. If it's outside of the requirements to have a protection against steady-state over-voltage conditions, that's fine, and obviously it's sufficient for your uses.

My remark was more towards the suggestion to use a polyfuse. These are really difficult to dimension correctly, especially when they don't operate under free air. A potted polyfuse may never blow, or very late, depending on how thermally conductive the potting material is.

Zener/SCR crowbars cope better with continuous over-voltage (you'll still need a fuse but it can be slow), but they are much slower than a TVS and not a replacement for transient absorption.

[KE5FX]

Great story.  It always sucks to have to haul a ton of extra 'just in case' devices and equipment/tools back from a trip where they weren't needed, but I'm sure you didn't regret it this time.  Good thinking on the rental car battery as well, I don't know if I'd have come up with that idea.  In your case I'd be tempted to implement radiolistener's approach with one (or maybe two) power FETs instead of the relay and SCR.   It should be possible to come up with a circuit topology that safeguards against both overvoltage and reverse polarity.

I've heard that forklift batteries are an underrated (reputation-wise) source of home energy storage.  Massive power capacity, deep-cycle compatibility, relatively low cost, and the companies that sell them are equipped to bring them to you and take the old ones away.

[tszaboo]

The word "calibration" comes to my mind.
There are two kind of equipment in a lab, calibrated ones and paperweight. Even if you dont cablibrate it with the same tests, that the manufacturer envisioned. Or with the same specification, that the original one.

[IDEngineer]

Great story.  It always sucks to have to haul a ton of extra 'just in case' devices and equipment/tools back from a trip where they weren't needed, but I'm sure you didn't regret it this time.

Indeed. I have a dedicated backpack for it now. All the mission-critical stuff goes into that and it stays with me as a carryon "personal item". If they lose my luggage I might show up in yesterday's clothes and smell funny, but the technical aspects of the meeting will proceed!

The one bugaboo used to be hand tools. TSA really hates those. I guess they're fearful you might disassemble the plane starting in the bathroom or something. But they specifically allow tools shorter than seven inches so I carry a few critical ones like tweezers, needle nose pliers, cutters, etc. in a six-inch-long ziplok. Sadly, that doesn't allow a large enough wrench to pull a battery from a rental car but nor did I think we'd need to do that!

I've heard that forklift batteries are an underrated (reputation-wise) source of home energy storage.  Massive power capacity, deep-cycle compatibility, relatively low cost, and the companies that sell them are equipped to bring them to you and take the old ones away.

That's true for electric forklifts, and when they first suggested using their forklift my first question was "what voltage are the batteries?". Most electric forklifts run packs in series/parallel and operate at some multiple of 12V. I hoped they exposed the individual packs so we could get to 12V. Fortunately, as mentioned above these were propane fueled forklifts so they had a traditional Group 24-style 12V starting battery, perfect for our needs. It was even conveniently mounted on a wheel transport so we didn't have to carry it across the factory! {grin}

EDIT, since you added something after my response:

It should be possible to come up with a circuit topology that safeguards against both overvoltage and reverse polarity.

Reverse polarity is covered nicely by a small series 1000V 1A Schottky diode. Nothing complex needed, we can afford the voltage drop, and sub-500nS response is plenty fast. Transients we have covered too. That leaves long-term overvoltage, and frankly it's not an issue in the real world. As I said, I've yet to see the lead-acid battery that accidentally generates 2X is chemistry's rated voltage. And our spec sheets are very clear about the max input voltage.

EDIT #2: I just finished characterizing our standard protection circuit for sustained high voltage conditions. We can safely tolerate a sustained overvoltage of ~18VDC before the tranzorb's conduction climbs enough to matter (above 10's of milliamps), and then the limiting factor is only internal heating of the tranzorb (proper device operation continues). That's fully 50% over the nominal rating of 12VDC, and 30% above the typical alternator voltage of 13.8VDC. If someone's on-vehicle regulator is permitting a sustained output voltage 30% higher than alternator nominal, ours won't be the only device that fails. In production there's always a limit to how much you can spend on protection against extreme cases while remaining commercially viable. I think we're OK where we are with 30-50% margin.

[IDEngineer]

The word "calibration" comes to my mind. There are two kind of equipment in a lab, calibrated ones and paperweight.

For Engineering T&M power supplies I would have the same opinion. But this wasn't anything remotely like an Engineering T&M power supply. This was a grunty high current fixed voltage power source. No adjustments were made available. I didn't crack the box but I'd bet there aren't any internal adjustments either - it's likely designed with fixed components and no "calibration" is required or even possible.

Next time I'm there, I'll ask if I can open it up. I suspect they'll be OK with that. I'll take photos of what's inside and post them here.

[radiolistener]

Reverse polarity is covered nicely by a small series 1000V 1A Schottky diode. Nothing complex needed

Schottky diode has voltage drop, for high current like 50-100 Amps it will dissipate a lot of power. Schottky diode is useful for a low power devices, with current consumption below 1 Amp.

[IDEngineer]

Schottky diode has voltage drop, for high current like 50-100 Amps it will dissipate a lot of power. Schottky diode is useful for a low power devices, with current consumption below 1 Amp.

The small signal circuitry this is protecting consumes <50mA typical. The 1A rating comes from our desire for the 1000V rating, given the magnitude of spikes that can occur in automotive systems due to field collapse. From a current perspective the diode is substantially overrated but a nice side effect is that at our 50mA it doesn't even get warm (as proven by our thermal studies).

[mawyatt]

I prefer the term "tranzorb" though most people think "TVS" so that's what I typed.

Our tranzorb's turnon time is, according to the spec sheet, 1ps typical (yep, picosecond). No (poly)fuse is going to come within many orders of magnitude. Remember, we aren't expecting the tranzorb to deal with steady-state overvoltage conditions. Its job is to absorb transients, hence the name.

Nice story, glad it worked out well in the end.

Transorbs bring back memories of the 70s when we were designing some equipment for controlling (synchronizing) the massive power generators in power stations. This equipment had to survive an almost direct lightning strike to the power grid, we used the Transorb to capture the leading edge of the wavefront, then an array of MOVs for the middle of the wavefront and followed by a ionization tube to dissipate most of the wavefront energy.

I recall long ago the Transorbs were originally developed by General Semiconductor under a USG contract. A story behind this development (can't say it's true) was a direct lightning strike hit an ICBM silo and caused multiple "Fail Safe" circuits to "Fail Unsafe"   Evidently back then the usual over voltage protection wasn't quick enough nor strong enough to deal with the lightning wavefront, and better protection was required.

Best,

[tszaboo]

The word "calibration" comes to my mind. There are two kind of equipment in a lab, calibrated ones and paperweight.

For Engineering T&M power supplies I would have the same opinion. But this wasn't anything remotely like an Engineering T&M power supply. This was a grunty high current fixed voltage power source. No adjustments were made available. I didn't crack the box but I'd bet there aren't any internal adjustments either - it's likely designed with fixed components and no "calibration" is required or even possible.

Next time I'm there, I'll ask if I can open it up. I suspect they'll be OK with that. I'll take photos of what's inside and post them here.

When you measure the output voltage, that is technically calibration. Int he original sense of the word.
"In measurement technology and metrology, calibration is the comparison of measurement values delivered by a device under test with those of a calibration standard of known accuracy."
You eventually did this, just in the wrong order.

[Bud]

so I asked for a wrench so I could go remove the battery from our rental car

Bet you could end up with a non-working airbag at a minimum, and had to deal with the car rental company to fix things.

[rsjsouza]

Great story, IDE. I had my share of faulty environmental issues caused by bad customer-furnished hardware, but nothing as dramatic. In my case it was malfunction that threw us in a loop until we sourced the root cause to things like ground loops coming from bad plugpacks, faulty USB or serial cables, misconfigured hardware, etc.

[radiolistener]

Bet you could end up with a non-working airbag at a minimum

why? I'm switch off the battery when the car is not used for a long time, it prevents discharge.

At standby mode it consume about 50 mA.
24 * 0.05 = 1.2 Ah every day.
30 * 1.2 = 36 Ah for month.

36 Ah is a 60% for 60 Ah battery.

[IDEngineer]

When you measure the output voltage, that is technically calibration. Int he original sense of the word.  "In measurement technology and metrology, calibration is the comparison of measurement values delivered by a device under test with those of a calibration standard of known accuracy."You eventually did this, just in the wrong order.

To me, "calibration" implies adjusting to spec - not simply measurement (unless the DUT is already in spec). If I measured this power supply and thus knew it was emitting 29V, yet left it in that condition despite its front panel clearly stating "13.8VDC", I would never describe what I had done as "calibrating it". Likewise, if I told someone I had "calibrated" that power supply and it was still emitting 29V, they would have a legitimate argument with me. One that might involve fisticuffs!

[Golds]

You are right!  I support your point of view.

[paulbt]

I whipped out the DMM    and measured the voltage coming out of their supply.

29.1 VDC.

      

Keep in mind this was a simple analog supply. Not a battery charger. Not a jump starter. It had typical threaded banana jacks on its front panel. It was meant for a bench environment. And it was labeled 13.8VDC right on the front panel, with no adjustments anywhere on its six outside surfaces.

Hi, IDEngineer !

Interesting story you shared to us, thank you! It is definitely a lesson worth to be learned or at least remembered from time to time. Good thing you brought with you a trusty dmm  , it saved you from total embarassment. 
About that power supply, if it was linear and beefy as you said, I think the pass transistor was shorted and the full voltage from the bridge rectifier was delivered to the output terminals (assuming it was a very simple design w/o crowbar or other safety circuitry).

Have a nice day! Cheers !

[Berni]

Nice adventure there. Love the MacGyvered forklift solution too.

Pretty sure i would have done the same mistake, if it looks like a fixed output PSU and it has the output written right there on the front id believe it and hook up to it. Surely they would have found out beforehand if it was faulty or something...right? Guess not.