SMPS PS-3381-1C1 Server power supply - five gone dead on power up

[eblc1388]

Friday the 13th, so what? Well. last Friday, 13th of July, five of my six SMPS decided to commit suicide.

In 2015, I've bought six used HP server power supply(PS-3381-1C1) and modified one of them into a 13.8V 30A battery charger, see photo. These power supplies would run the internal cooling fan at low speed once powered up. When the 12V output is enabled, the fan would run at full speed. All the six power supplies did this when I bought them. The remaining five I wrapped up carefully in bubble wrap and kept for spares inside my study. The modified charger worked happily for about six months and then I stopped using it and it was sitting on a shelf.

Last Friday I powered it up and it worked for several seconds, then it shuts down. It wouldn't startup again, no smoke, no sound, no smell, nothing. Bad luck, I thought, no big deal as I still have five spares.

I took out a spare one and powered it up. Instead of the cooling fan running I got a faint short click or pop sound. It wouldn't startup. I took out yet another one and this time I got lucky. The cooling fan ran on power up. But the happiness was short lived. It went dead several seconds later. It wouldn't run again. Now three of them were dead upon power up for unknown reason but fear not, I still got three more.

By then I have realized something is not quite right. I measured my AC mains voltage and it was 247V while the SMPS supply nameplate voltage is 240V. Could the high mains voltage be the culprit? I took out a variac and set its output to 220V. I powered the fourth PS with 220V but it also failed like the second one, given out a faint pop, no smoke or smell. Four have gone, two to go.

With only two supposedly good power supply remaining, I cannot carry on as usual. I needed to be extremely careful. Then I remembered the golden rule in SMPS testing. I placed a 60W filament lamp in series with the 220V supply and powered up my fifth supply. The trick worked. The cooling fan was running happily. I measured the AC input to the power supply after the lamp and it was only 60~70V. I left the supply running for about ten minutes more "just to let the circuit settle down". I thought I've finally beaten Murphy. Well its time to remove the lamp and powered it up directly. Wrong!!! The dreaded faint pop rears its ugly head again and I'm left devastated. I vowed never to power up my last one until I can find the root cause of the incident.

With the help of the "healthy" SMPS, I quickly located the damaged parts on all five defective SMPS. The schematic is shown below. The top part is copied from the datasheet of the TOPSWITCH which I have annotated with the actual part values(with pink background) used. The SMPS designer had added his own circuitry(within dash lines) at the bottom.

The TOP223YN PWM switcher, which powered the auxiliary control circuit and the cooling fan, failed with a short between "Control" and "Source" pin. The "Source" and "Drain" is opened in both directions. All five TOP223 failed in the same way. Nonetheless, the PCB mounted 1A fuse in series between the high voltage 330uF capacitor and the switching transformer was intact. I removed the TVS diode 1.5KE200A and tested it using DC 500V in series with a 100K resistor. The breakdown voltage across the diode is 198.5V which matches closely its specification.

A mystery SMD component marked "A071" / "A084" failed short too between pin#2 and #3. This SOT-23 component was mounted on the underside of the circuit board so there is no silkscreen marking to indicate if it is an IC or just a plain semiconductor. DVM Measurement on the "good" SMPS shows readings similar to a NPN Darlington transistor but as the base is grounded it seems not plausible to me. Could it be a SCR?

Well, I would never trust this SMPS again even if I can repair it. I'm doing all these to satisfy my own curiosity as I've never in my life been in such an awkward predicament.

My questions are:

1. As seen from the schematic, what can be said about the component marked A071 or A084 and its possible function?

2. A plausible reason why all the TOP223YN failed in a properly designed power supply?

3. I have noticed an unusually low C5 10uF capacitor value at the TOPSwitch "control" pin input, which nearly all data sheets suggested should be 47uF. Could this low value capacitance leads to the demise of the TOP223YN?

4. I have some TOP223YN ordered. What steps should I follow to troubleshoot the SMPS if it works with a lamp load in series after TOP223YN replacement?





 

 

[Relayer]

Hello eblc1388,
I'd say that "AO71" is a voltage reference IC that supplies 1.5V at Pin1.
There's also a good chance that IC600 is used as a crowbar protection device.
With each case, especially since you heard a pop in several of the units, was the
1A fuse blown?
I would check R4 and R5 for any drift in their values, as they have to be
fairly accurate to supply the correct amount of feedback to the opto-coupler.
Regards,
Relayer

[eblc1388]

I'd say that "AO71" is a voltage reference IC that supplies 1.5V at Pin1.

Judging from the voltage divider configuration at pin#1, it is more likely "A071" is some kind of checker, monitoring the high voltage level on the 330uF high voltage capacitor.  It is either a low voltage inhabit or a high voltage shutdown, but I'm not sure which without knowing what the part is.

I can't locate any information on this SMD part, so it could be a proprietary part. With a leap of imagination, it could even be a part which is designed to "terminate" the life of the supply, after a long period of inactivity.

There's also a good chance that IC600 is used as a crowbar protection device.

The TOP223YN is a common three pin switcher which incorporate the main power MOSFET and all the control function inside the chip.

With each case, especially since you heard a pop in several of the units, was the 1A fuse blown?

The 1A fuse was not blown. This led me to believe that the TOP223YN was damaged via high voltage or high current fed into its control pin.

I would check R4 and R5 for any drift in their values, as they have to be fairly accurate to supply the correct amount of feedback to the opto-coupler.

I agree to your view if it was only one unit being damaged. But it is difficult to convince oneself that all five units have the same drift in resistor values.

[Relayer]

Hello eblc1388,
Thanks for clearing up my assumptions. I guess I was right off the mark. 

I agree to your view if it was only one unit being damaged. But it is difficult to convince oneself that all five units have the same drift in resistor values.

I remember when I was doing computer monitor repairs mid 1995, we had one brand of 14" monitor that one of the
start-up resistors kept going high to the point that it destroyed the main chopper transistor, opto and blew the fuse.
Every time we had one of these come in, that's the first component to be checked. The company I worked for made
a fair bit of money due to this particular resistor. Must have been a shoddy batch from the resistor manufacturer.
Regards,
Relayer

[eblc1388]

Well, my TOP223YN switching regulator spares arrived and I managed to bring back one of the failed units to life. I still have no idea what the unidentified A071 SOT23 chip is so I just leave it out. So far it doesn't seem to affect normal operation of the SMPS.

Now comes the fun part. I still have one "healthy" unpowered unit which I left alone till this very moment. With one repaired spare unit available I can now afford to power it up and see if it failed in the same way. Usually Murphy would balk at my idea and make the unit works normally. However, the temptation is so great I have to try.

I connect up my DSO channel one to the PFC output voltage and CH2 to the drain pin of TOP223YN. I set up for single trigger and double checked my connections as I have only a single chance if the unit fails.

Luckily, Murphy was off somewhere and the unit failed as expected. Have you seen someone rejoice when thing fails? Now I have a recording of when in the starting process the TOP223YN failed but not a reason. The SMPS is operating without any loading on its +12V and +5V outputs. The PFC output voltage is 380V and TOP switch absolute maximum breakdown voltage is 700V. It looks as if the spike had exceeded the breakdown voltage of the TOP switch and caused its failure.

Perhaps someone can offer some insights into the waveform and the failure mechanism.

I also attach two image regarding the insulation of the inductors. The first one is the common mode choke and the other one is PFC inductor with an auxiliary winding. They look scary to me.

[siggi]

TVS1/D1 should be clamping the switching spike to Vpfc+~200V. I’ve seen TVS die of apparent fatigue in similar circuits. Is that your problem here?


Sent from my iPad using Tapatalk

[eblc1388]

TVS1/D1 should be clamping the switching spike to Vpfc+~200V. I’ve seen TVS die of apparent fatigue in similar circuits. Is that your problem here?

No, it is not.

I've somehow figured out a possible explanation after looking carefully at the failure capture waveform. During the initial starting process, the secondary voltage regulation control circuit for unknown reason(or powering up instability in its control loop) requested an increased duty cycle from the TOPswitch. This caused the PFC output voltage to sag. The PFC circuit then trying hard to compensate this increased loading by raising the output voltage back to the nominal value while the regulation circuit simultaneously decreasing the duty cycle of the TOPswitch, causing the PFC output voltage to spike. Although the TVS diode is clamping the pike to below 250V above the PFC output voltage, its total magnitude is still too high for the TOPswitch so it broken down.

All my other faulty power supplies were repaired after replacing the broken down TOPSwitch TOP223YN.

[z]

Check the 2 Ohm resistor shown in the photo.  ML4800's VCC voltage  comes from this resistor.  If this resistor burns, power supply goes  off.

[charlyiy]

hi mr eblc1388, im aware this is a old post.. but i just got one of this PS-3381-1C1 power supply, and i want to make the 13.6 mod to this one... will you give the guide to do that? i will be gratefull for that... im not a englis speaker so mine would be a little confiusin...  tnks on advance....

[eblc1388]

... will you give the guide to do that?

It involves two steps. First is to increase the voltage adjustable range to about 14.0V so user can trim the output voltage to 13.6V or 13.8V, using a 68K resistor. However, the internal over voltage protection will triggered so we have to take care of it in the second step. The additional 6K8 resistor will increase the trip point to 14.45V.

The attached image will show the before/after of the modification. You can use regular 0.25W resistors too.


 

[charlyiy]

big thanks to you... im about to mod this as soon i get home.. at afternoon.. will give you some feedback then.... txz again..... and by the way.. im gonna run this one at 110v not 220 as u does.. hope it keeps it working.. maybe something in that voltage that get yours down..... tell you later... just a question... how i trim the voltage?

[eblc1388]

The output voltage can be trimmed via the small variable resistor on the small circuit board. See picture below:

 

[jessoman]

I can confirm that the topstitch is a common failure point. I'm a dumb bugger at electronics. But it only took about 10 mins to switch over as a n00b.
Pow4 💅

[eblc1388]

Now my six PS-3381-1C1 once dead but all repaired successfully by replacing the TOP223YN switching regulator.

I have come to the conclusion that the failure was manually introduced during the refurbishment stage, before being sold on ebay. The internal of these power supplies were cleaned using some forms of cleaning solvent and somehow the solvent ingress into the regulator's pins. Gradually the damage was done to the TOP223 internal circuitry after a year or so.

It can be the only plausible reason why a batch of six working power supplies, refurbished and then working fine but all failed a year later on startup.

[45Overload]

Thanks for documenting the repair of these supplies.   I think the post demonstrates some common ways to jump too many steps ahead on any analytical process.  Very interesting that we eventually learn that all these units were refurbished by an unknown ebay seller.  What was the manufacturing date of the supply - how many years had these supplies been in service? No doubt, the seller purchased lots of these blown supplies and found a way to fix them (temporarily), but we can't know that.  We don't know if HP made faulty design choices.  All that is known at this point is that the supplies failed on power-up one year after purchase/refurbishing, and that replacing the switching IC brings them back to life.

-Were the supplies put into service after being repaired?  Have they been working fine since 2018? 
-Various other explanations could fit, such as installation of counterfeit parts by the ebay seller
-If an aggressive cleaning solvent was used, then a lot of parts on the pcb could have been damaged, not just the switching regulator
-In the world of switching power supplies, the switching transistor/IC is the most common part to be destroyed (along with failed secondary filter caps), simply because it must withstand high peak currents and high reverse voltages.   In many cases the real fault is elsewhere in the supply.  Lots of fixers think that their supply is completely repaired after replacing one faulty part. 
-All the parts in the supply age and change value at unknown rates.  As the parts slowly change value, lots of operating parameters can move beyond acceptable range.  At some point the supply cannot close its negative feedback loop.
-Interesting description of a computer monitor known to have one defective resistor.  Were those resistors somehow defective from the resistor factory?  If so, maybe a lot of resistors on that product were also out of tolerance.   Some shops might replace the resistor and be done with it, but there is always more to the story.  You don't want to be replacing the same resistor multiple times in the same unit. 
-A final thought about tweaking the output voltage of a switching supply.  This is not the same as changing the voltage reference in a linear supply.  A switcher is more like an RF transmitter.  Altering the PWM signal will essentially run tuned circuits at some unknown frequency or pulse width, likely generating excess heat, etc.  It obviously works ok in some cases, but it isn't that simple.
-45O

[eblc1388]

What was the manufacturing date of the supply - how many years had these supplies been in service?

There isn't a date printed on the PS-3381-1c1 big supply rating label except a small label saying "REV:06, B0452". I guess we can trying to interpret it to mean the 52th week of 2004 as the production date. As they were retired from some working environment, I would guess they had served three or more years in service. 

No doubt, the seller purchased lots of these blown supplies and found a way to fix them (temporarily), but we can't know that.

Certainly not. I can confidently say that the supplies were working fine when I got them and they have NOT been opened for repair before. The TO-220 regulator is very difficult to replace and would take a lot of effort. During my repair on these units, I found all the soldering joints looked pristine from original wave soldering. The seller I got them from have hundreds of them for sale so it is unlikely someone would repair these and then sold them on for a few pound/dollars each.
 

Were the supplies put into service after being repaired?  Have they been working fine since 2018?

Yes, all are still alive and one is in constant use to this day. 

In the world of switching power supplies, the switching transistor/IC is the most common part to be destroyed (along with failed secondary filter caps), simply because it must withstand high peak currents and high reverse voltages.   In many cases the real fault is elsewhere in the supply.  Lots of fixers think that their supply is completely repaired after replacing one faulty part.

I agree to your view in general but the supplies were retired from working environment before being sold for a few bucks each. what is the chance of each and every switching regulator failing upon power on after working fine in the previous year? Even doggy regulators will not fail in this manner for all six of the supplies.
 

A final thought about tweaking the output voltage of a switching supply.  This is not the same as changing the voltage reference in a linear supply.  A switcher is more like an RF transmitter.  Altering the PWM signal will essentially run tuned circuits at some unknown frequency or pulse width, likely generating excess heat, etc.  It obviously works ok in some cases, but it isn't that simple.

What you said is certainly correct for large percentage of output voltage changes like 20% or more. However, most tweaking related to server power supplies are limited to within 10% of the nominal output voltage. Nearly all of them perform their output regulation via changing duty cycle of a fixed switching frequency. Except a few cases, results indicated that most of the tweaking work fine, while some could face triggering the overvoltage protection. If you think about it, the PWM pulse duty cycle to produce 12.3V full load output could well be larger than of 13.6V at light load.

[nightfire]

In my experience of approx 20 years working with lots of servers, I can say that usually Powersupplies go dead upon getting cold, unplugged from mains and then have to do a cold start afterwards. Probably also aged electrolytics contribute to that issue, as they draw more power they could handle and whether pop or damage other components.

This sometimes happens after some maintenance work, when servers have to be switched off and are allowed to go cold for about an hour or so, or when a cold spare unit is powered up for the first time after one year or more in storage...

[eblc1388]

Nearly every server power supply design will have the main output PWM converter and possibly the PFC circuit disabled during cold start, until it is instructed by the server to provide the main output. During cold start, only the standby circuit is powered and in control. Thus the start up stress will fall mainly on the standby circuit and its switching regulator only. As most failure occurs in this part of the circuit, this could easily be the weakest link in the whole of the supply design.

 

[jessoman]

This is a interesting discussion.

After playing with my PSU's today. I tried the load share feature (pin 19 for memory, joined together on 2 PSU's). To charge a large lifepo4 battery. I could only get max 30 amps. I thought if I put them together it would allow 60.

Or is it 30 total regardless, and the only way to get 60 would be to diode the outputs and disconnect the load share?

Apologies if this is real n00bish

[eblc1388]

Say with only one power supply the battery is drawing 30A, then current sharing with another power supply would result in the same current draw, i.e. 30A but now shared 15A each between the two supplies.

The battery itself will ultimately determine how much current it will take from the two supplies.  If the battery terminal voltage is very close to the power supply output voltage, then you cannot get more current into the battery regardless of how many supplies are in parallel. However, if the battery is almost empty, you may get more charging current into it.

[jessoman]

Ah, problem solved. Cable too small and didn't check the voltage at the battery. 70 amps in. Not sure why over current protection didn't kick in. I'll drop the voltage. But this is great for me. These little guys are beasts.

[45Overload]

Thanks for the comments.  I'm glad you put one supply in service and confirmed that it continued to work ok.  Certainly when a supply is barely kicking along you will often find it won't power up again - usually due to electrolytics that warm up due to high ESR and provide a  minimum capacitance when they are warm.  Another related topic is soft start which is a fantastic field by itself.  Of course it costs more to implement,  but necessary when the inrush current is massive.  Some switching supplies won't start if you apply voltage gently with a variac, so testing with a current limiting device (light bulb) is needed. 

Let me generalize briefly because troubleshooting in general is very interesting to me.  I divide the topic into 2 very different parts.  The first approach is to rely on personal experience.  This method does not require an accurate problem description.  Repair or system implementation is done via intuition.  Nothing wrong with applying experience to your technique.  Lots of stuff gets solved in this way.   

The second type of troubleshooting is based on method.  The first step in having a method requires an accurate and complete problem description.  Method is the key to getting stuff fixed that refuses to yield to experience.  Method always works.  It may take a long time and lots of notes, but you will always get there in the end.  Teaching method to younger engineers is difficult but important.  One problem with getting carried away in the method category results in "over-teching" a simple problem, so experience does enter into the whole equation.

One approach that I don't know what to do with came up when a guy I worked with had to repair a sophisticated video processing unit.  3 or 4 other guys including myself could not make any progress in understanding the design.  Nobody could think of an approach that would provide some valid information.  After a few days guy #1 announced that he had fixed the video unit.  We gathered together expecting to get a good lesson on design and repair approach.  This unit was instead "fixed" by replacing whatever spare parts were in stock at that time.  Good result.  Strange approach.  Stuff does get fixed this way but generally nobody is any the wiser. 

That's all I have to say on the topic.
-45O