Repair Help for APC UPS BR1500G - F06 error (FIXED!!)

[fmashockie]

Hi everyone! New to this forum but I've always been a huge fan of the EEVBlog YT channel!  I'm trying to repair an APC UPS BR1500G.  I had it in storage for a few months (with battery installed unfortunately) and took it back out to test and it started triggering a F06 error.  It would start-up fine, go through a self-test (switching to battery power) and then switch back to mains and trigger this error.  According to the manual, F06 = relay welding.  I've tested all the relays out of circuit and none of them have stuck contacts.  But I replaced them all just in case and it did not resolve the issue (couldn't be that easy  ).

I found that when forcing the unit into Automatic Voltage Regulation (AVR) mode, the unit will not trigger this error.  But only at increased voltage (>130V).  I can simulate a power failure at this voltage and it will switch to battery power without issue.  I can also put a load on it and it will handle it fine.  But as soon as I bring the voltage back down to normal levels, it will trigger this error again! (Quick note - If I attempt to start-up the unit at a lower than normal voltage like <110V, the unit automatically starts in the F06 error - no self test, etc.)

One thing I noticed while the unit was in AVR mode is that there is a significant ripple that appears/disappears at regular intervals on the output.  I observed this while hooking the output up to my oscilloscope.  I have a video of it here () but it is a significant ripple/noise (~10V peaks). 

I would suspect that the transformer is the problem - (maybe the 130V tap has internal breakdown?), but I did a HI-POT test on the transformer up to 1000V and it passed the test.  I've tested similar transformers with internal breakdown and they fail immediately. 

Anyway, I was hoping someone might have some troubleshooting tips/suggestions/help!! This repair is pretty much academic at this point.  It's only a $200 UPS (one of APC's most common offerings).  But I'd love to learn more about how to troubleshoot these things.  Thanks!!

[fmashockie]

Is there any additional info I can provide to help better understand the issue? Just lemme know! I would appreciate any help.  Not looking for someone to fix this for me, just how they would approach this problem (at least). 

[Vovk_Z]

How old is it? Old ones tend to want to replace small electrolytes which can cause all types of problems. I'm not sure here is such a problem but who knows.
Have you measured output voltage when you regulate input voltage? Is it within range? Does it change as should?

[fmashockie]

Thank you for the reply! I did test the electrolytic caps in-circuit with an LCR meter.  For the ones I could measure in-circuit, they measured normal.  But couldn't hurt to measure again and even measure some out of circuit.

The output voltage stays within range when adjusting the input voltage.  For example, in the video I took in my OP, the input voltage is set to ~130VAC.  But output measures ~120VAC. 

What I am trying to figure out is why the unit will only work when voltage is increased and the unit is in buck mode.  If I start it up with mains voltage at normal 120VAC (I am in the US), it will do its self test (switch from mains to battery and back to mains).  Then I hear another relay click, and I see increased noise in the AC waveform, and it triggers this F06 error and shuts down.  If I start the unit with a less than normal voltage, (say 110V), it will automatically go into this F06 error.  The error means relay welding according to the manual, but the relays are fine!

[fmashockie]

How old is it?

Sorry I forgot to answer this question.  It is ~5 years old.  It was in service 24/7 for ~3-4 of those years.  Then I took it out of service and put it into storage.  There was nothing wrong with it when I took it out of service initially.  I did however store it with the battery installed.  The battery appears fine - voltage measurement indicates its in good shape.  The unit can power a load (~100W) on battery power.  It can also run in battery back-up mode just fine as well.

[RLP]

Hi fmashockie,

We know for sure that the UPS thinks that one of the relays is faulty - but there may or may not actually be a relay problem. Based on my experience with a different APC product, the UPS may be detecting AVR relay weld faults by measuring its input and output voltages. The buck/boost operation changes the ratio of the AVR transformer to buck/boost the output voltage relative to the input voltage. The UPS can't actually detect what the relays are doing directly, but it can look at the ratio between the measured input and output voltages. If that is not as expected, it might trip the fault - essentially assuming a relay is stuck in the wrong position. So if one of its voltage measurement circuits is faulty, it may incorrectly detect a fault.

However, you say that it outputs 120V with an input of 130V... that would suggest to me that its voltage measurements are OK, because it's doing the right thing under those conditions at least.

If you have replaced all the relays (and the old ones tested fine anyway) and it's still doing this, then I would suspect that one of the relay driver circuits is faulty.

Why would it only work at high input voltages and fail otherwise? That seems like an important clue as to which relay is failing. Based on that, I'd say the AVR trim relay is not working, so it's basically stuck in voltage trim mode. As soon as you drop the voltage to a point where it would usually switch out of trim mode, it tries to and fails, and trips the fault.

I guess you'd have to figure out which relay is the trim relay, find the drive circuitry for that relay, and troubleshoot it. I'd start by looking at it, in case anything is visually blown - and then compare its input signal (from the microcontroller or other circuitry) with its output to the relay coil, to see if it's working as it's supposed to. A schematic would help a lot with that, which may be available online somewhere.

[fmashockie]

Thank you so much for this detailed explanation RLP! I really appreciate it.  This is kinda the route I was going to go with.  Interesting to think that the AVR trim relay is the problem.  I could have sworn when I had it in AVR mode (buck) that when I began turning the input voltage down, I heard the trim relay click, and then the F06 error was triggered (as I show in the video), but I could be wrong.  But regardless, I get your thought process and will begin with that investigation once I get back in the lab tomorrow.

Do you have any thoughts on that noise/ripple I showed on the oscilloscope?  I feel like that cannot be normal. 

Thanks again for your reply!!

[RLP]

In the video, I couldn't hear any relays click when the AVR icon disappeared from the LCD - only a click as it later shut down with the fault and started beeping. I would expect a click just as the AVR icon disappears.

For the ripple/noise... the way it disappears and comes back reminds me of the UPS's internal battery charger, which goes off and back on briefly to test for the presence of batteries every so often. You have the bare board quite close to the oscilloscope and probe cable - I'm wondering if maybe it could actually be EMI from the board, maybe the battery charger, affecting your measurements? I imagine that level of ripple, if actually present, would be audible in the input/output filter components - can you hear that frequency audibly anywhere?

These UPSs can certainly make for a bit of interesting diagnosis/repair!

[BradC]

I had an SU2200 which started giving intermittent relay weld faults. I too replaced the relays. On the older SU series, both input and output voltages are measured by little PCB mount transformers, and the transformer on the input had developed a perverse fault in the primary that wasn't quite an open circuit, and was strongly influenced by humidity and temperature. I replaced that with another from an older scrap unit and it's been good since.

In addition to the relay drive circuits I'd look at the input and output voltage sampling. I was lucky as with the older units I had a schematic available.

[fmashockie]

These UPSs can certainly make for a bit of interesting diagnosis/repair!

Yes they can!! Thanks a lot for the input RLP I really appreciate it!  It might be some time before I get back to you with an update.  This is probably the lowest on priority of list of things to do at work (unfortunately it's the most interesting ).  I'll try narrowing down where that noise is coming from - try moving the board further away from the scope - make sure there's no interference. 

I had an SU2200 which started giving intermittent relay weld faults. I too replaced the relays. On the older SU series, both input and output voltages are measured by little PCB mount transformers, and the transformer on the input had developed a perverse fault in the primary that wasn't quite an open circuit, and was strongly influenced by humidity and temperature. I replaced that with another from an older scrap unit and it's been good since.

In addition to the relay drive circuits I'd look at the input and output voltage sampling. I was lucky as with the older units I had a schematic available.

Thank you BradC.  See that is what I was afraid of hearing.  This unit has little current transformers as well to sense the load (maybe voltage, too? I assumed that was done by resistor ladders).  I had worked on a similar model APC UPS in the past (the BR1500MS which is pure sine-wave unlike this one) and it kept throwing errors and blowing the inverter MOSFETs.  It turned out the large AVR/inverter transformer was the cause - must have had some internal breakdown.  I HI-POT tested it and it failed immediately (I have a video of that on YT as well).  Dr. Carlson's Lab also does a video on this APC unit and discovers the transformers might be made of aluminum (not copper) and his tap connections were crimped very poorly.  My issue was not that easy to discover.  But in short, I'm saying I'm not liking APC's track record with cheap transformers.  Transformers can be the most difficult component to replace.

And unfortunately, no schematic for this model.  I've looked everywhere.  I've found schematics for older models, but not for these newer models (like this one) that are currently available for purchase.   

Thank you both again for your input! I'll be giving an update as soon as I can take another look at this thing!

[Vovk_Z]

I tend to double RLP thinking, I tottally agree. It seems to me that main signal processing unit is fine but some output circuit is faulty, e.g. a transistor controlling that AVR relay etc. There possibly could be a burned wire too (because of malfunctioned relay etc).
There is a possibility that TS soldered wrong type of relay because they are different. They are not one type but two types I guess (NO and SPDT ?).

[fmashockie]

Thanks for the reply Vovk_Z.  The relays I used as replacement are a different make/model than the originals because they were no longer available.  But I matched them exactly to spec.  All 7 of them are form C relays (both original and replacement).  And it was having the same issue before I replacement the relays.  Which is why I replaced them in the first place lol.

[fmashockie]

Just a quick update.  I had some time to work on this thing yesterday.  I might have found a suspect transistor involved in the switching of one of the relays.  It is a internally-biased SOT-23 package.  There are several of the same type on the board and it was giving a much lower forwad voltage on diode test than all the others.  Unfortunately, it broke apart as I was trying to remove it with my heat gun (maybe a sign it was in bad condition).  I've ordered some replacements so it will be a bit until I can work on it again.

I also removed both of the current transformers to test them.  Both of them had normal resistivity/continuity readings (~0-1 ohms on primary, ~100 ohms on secondary).  And I HI-POT tested them and each of them passed up to ~500-1000V (didn't want to crank it up too high as they are small little guys).  There's also another large transformer on the board by the inverter which I'm not quite sure what its purpose is.  The pure-sine wave APC UPS I worked on in the past didn't have one of these.  But I removed/tested it anyway and it seems fine as well.

Also tested all electrolytic caps in-circuit again and none of them seem suspect. 

A side note - it is a real pain to work on these UPSs.  They use very large traces/planes due to the large amount of current/power they have to be able to handle (up to 1500VA).  Desoldering/soldering anything on them can be difficult.  Have to employ my heat gun and soldering gun at same time.  A few years ago I grabbed a Quick 861DW after seeing Dave do a teardown on one.  Works really well!

Thanks again for everyone's input!

[fmashockie]

Just wanted to give an update regarding this repair.  So my company just got in two brand new UPSs.  They are BR1000MS2 models - APC's pure-sine wave offering at ~$200.00.  It's a bit different than the one I'm trying to fix since the BR1500G is a simulated sine-wave model.  But I decided to open this brand new one up to see if it could help me troubleshoot.  I forced it into AVR mode by powering it from my Sencor PR57 transformer at ~130VAC.  I then probed the secondary side of the large inverter/AVR transformer and found that voltage on outer taps was = 130VAC and the 'center' tap was = 115VAC. 

I realized that this was about the readings I was getting for the BR1500G even when it was not in AVR mode (i.e. >130VAC input voltage).  For example, even when on battery power, the BR1500G would measure >130VAC and ~115VAC on outer and center taps, respectively.  On the BR1000MS2, on battery power the outer and center taps measured ~120VAC and ~110VAC, respectively.

So with that said, I'm beginning to think the issue is not with the any of the relays after all.  But that there is a problem with the charge/inverter circuit.  Also, there's no noise observed in the AC waveform on the output voltage for the BR1500MS2 unlike the BR1500G.  This would also explain why the BR1500G does not trigger the F06 error in buck mode since it is expecting those secondary outer and center tap voltages. 

If it is a problem with the inverter/charge circuit, could anyone explain/speculate what could account for the increased voltage?  As I said, I clearly recall measuring close to ~140VAC and ~115-120VAC on the outer and center taps of the AVR transformer without it being in AVR mode.  It was in battery mode during these measurements yet still providing normal output voltage (~120VAC).  I'd be curious to know anyone's thoughts on how that could happen.

I'm still waiting on replacement components, but will provide an update once I receive them and install.  Here's a link below to a video I did explaining my observations.  Thanks!

[RLP]

Interesting. I think there may be some topological difference between the BR1000MS2 and BR1500G, in terms of how the inverter and transformer(s) work. The BR1000MS2 looks to me like a low frequency inverter, where the big 50Hz transformer is part of the inverter and thus has large high-current terminals on the low voltage side coming from the MOSFETs, and various taps on the high voltage side. I can't tell for sure but the BR1500G looks like a high-frequency inverter design: the transformer on the PCB looks like part of a high-frequency DC-DC boost converter used as part of the inverter (it's too big to be part of the battery charger), and so I suspect that the big transformer is probably only used for AVR. I can't see any big high-current connections to that transformer in your videos - for a 1500VA UPS, there should be some large high-current terminals on the transformer if it steps up the battery voltage to mains voltage?

My thinking is, maybe there could be some legitimate topological reason for the difference in voltages while on battery?

Then again if a relay is stuck (due to drive circuitry etc.), that could also cause the voltages to be different - but (maybe) it may not matter or be detected while on battery power.

I'll be interested to hear what you find after replacing parts.

[fmashockie]

Hi RLP.  Hmm that is a very good point.  I just assumed that with the BR1500G that the large transformer was being used for both the inverter and AVR just like the BR1500MS2.  However, the BR1500G is the simulated sine-wave model and the BR1500MS2 is the pure sine-wave model.  And I've always wondered what that large transformer on the PCB was for so I think you might be right.  I suppose is it is still too early to be jumping to conclusions.  It's just with the exception of that internally biased BJT (still not sure if it was bad or not), I didn't really find anything that stood out on the controller side for the relays that would indicate a fault.  And even though they employed a lot of cheap electrolytic caps (OST, CapXon, Chang, etc), none of them really measured that poorly once I got them out of circuit.  Still replaced them though.  Can't wait to get these remaining parts in to see what happens.  I just thought it was interesting to share.  Really appreciate your input on this.  Thanks again!!

[fmashockie]

The BR1500G saga continues.  I received all the replacement components I ordered (all the MOSFETs and that internally biased NPN BJT by one of the relays).  I soldered them all in and fired the unit up and of course... it's still having the same issue  When the input voltage is ~120VAC, I start it up, it does its self-test (switching from battery to mains and back), and then I hear another relay click, and then it triggers the F06 error.  After that, I increased the input voltage to >130VAC to force into AVR mode, and it was working without triggering the F06 error.  But as soon as I brought the voltage back down, the error was triggered. 

But then something else happened.  I was running it in backup battery mode (using my DC supply to limit the current) so I could make some measurements.  The unit will beep every 30 secs or so to let the user know its on battery.  All the sudden, the buzzer got much louder.  I decided to check the input voltage and supply voltage on a nearby LM317T and LM358 and they had increased by ~10VDC!  It stayed this way for ~1 min. Then the buzzer got lower, and all the sudden the unit powered itself off.  I couldn't get it to start back up via the display controller after this happened.

I took some measurements. The ~24V rail (it usually sits at 26-27VDC) was ramping up and down by ~1V every 500msec or so.  The 3.3V rail was doing the same.  I have attached photos of the waveform for these measurements from my oscilloscope.  Any thoughts on what could be causing this? Or where I should begin to look for faults? I'm not sure if I've caused another issue or just made the original worse (i'm hoping its the latter).  Thanks in advance!!

[Vovk_Z]

I'm not sure 3V3 rail is ok. Are you sure you are doing measuring right? It mustn't have such a pulsations I guess. Are you sure you have replaced all electrolytic caps?

[fmashockie]

I'm not sure 3V3 rail is ok. Are you sure you are doing measuring right? It mustn't have such a pulsations I guess. Are you sure you have replaced all electrolytic caps?

Yes I know the 3.3V rail is not okay.  None of the secondary power rails are okay.  They should not be ramping up and down like this.  And it is definitely a problem because now the unit will not even turn on.  That's why I was asking for help!!  And yes I'm confident I took the measurements correctly.

[RLP]

From your description I'm thinking an onboard power supply failed in some way, and put out an abnormally high voltage, as you measured - and then some downstream component, maybe an IC or another power supply, failed due to that overvoltage. The failed component probably went short, hence it all shut down and won't power up. The ramping behavior you see might be power supply 'hiccuping' where a power supply is shutting down due to overcurrent, then restarting, then the cycle repeats. 500ms to a couple of seconds sounds about right for that.

If that is the case, I would say you need to find the shorted component. One idea is to try powering it up again and see what gets hot. Another approach is to measure the resistance of various rails to ground (3.3V, 12V, etc.) and see if one is obviously shorted, and then inject a controlled current into that rail from an external power supply and measure voltage drops across PCB traces on that rail to locate the fault.

If you remove the shorted component and power it back up and there is still a power supply issue, the overvoltage condition might come back and cause more damage! So the root cause is important too. This is kind of familiar to me because I had a similar thing happen when I was working on a UPS once. In my case it was the main 12V power supply of the unit, which went up to about 25V! It turned out to be because I'd damaged a resistor in the feedback loop of a switching regulator, while soldering a probe to it to measure the voltage, and it was a complete distraction from the main problem I was troubleshooting on the UPS. I had to replace a few ICs and fix the power supply - but it was alright in the end.

Good luck!

[fmashockie]

. This is kind of familiar to me because I had a similar thing happen when I was working on a UPS once. In my case it was the main 12V power supply of the unit, which went up to about 25V! It turned out to be because I'd damaged a resistor in the feedback loop of a switching regulator, while soldering a probe to it to measure the voltage, and it was a complete distraction from the main problem I was troubleshooting on the UPS. I had to replace a few ICs and fix the power supply - but it was alright in the end.

All too similar to what I'm dealing with here!  Thank you so much for your help RLP.  Unfortunately, I don't see any obvious shorts yet.  But definitely will be taking a closer look!

I'm not sure 3V3 rail is ok. Are you sure you are doing measuring right? It mustn't have such a pulsations I guess. Are you sure you have replaced all electrolytic caps?

My apologies Vovk.  It turns out I was not measuring the 3.3V rail in that second oscilloscope photo.  I did however find out why the controller was not turning on.  The DC/DC converter (MC34063EC) was not switching.  I measured no voltage on the emitter switch pin 2.  Not sure why the converter is not switching tho.  When it is not switching, it does not provide voltage to an LM317T and a LP2950 (KY5033).  The KY5033 is the regulator for the 3.3V rail.  I supplied 3.3V to the output and the display came on. 

I decided to desolder the MC34063 to see if the hiccup on the 24V rail would stop, but it did not (see attached photo).  So still more troubleshooting left to do.  Thanks for everyone's input!

[Vovk_Z]

  I did however find out why the controller was not turning on.  The DC/DC converter (MC34063EC) was not switching.  I measured no voltage on the emitter switch pin 2.  Not sure why the converter is not switching tho.  When it is not switching, it does not provide voltage to an LM317T and a LP2950 (KY5033). 

I totally lost what are you doing and what is happening. There was only one problem at first - a problem with a relay. But now you have more problems? 
I don't know what to say because the more you are into it the more problems you have. 

[coromonadalix]

and without a proper schematics, at some point you only do guessing ... and create maybe more problems than needed

on some 1500 models i did saw  some mentions about smt alu caps creating more noise on the supply lines ... and one in a feedback loop




i had one of theses who almost blown up the battery cells, had problems getting them out     trashed the unit

[fmashockie]

Yep it seems that could definitely be the case here.  No schematic for newer APC models like this one - I've looked everywhere.

At this point, my hopes of fixing it are not high.  I'm just hoping to learn from the experience and troubleshooting.  I'm self-taught with this stuff and mostly learn by doing.  I definitely appreciate everyone's help though!

[fmashockie]

Still haven't quit on this thing yet! So as I said, for a bit there I thought I had killed this thing for good.  But I found out the reason why the LCD controller was not powering up - I must have inadvertently fried the DC/DC converter.  I replaced it and it powered up fine.  But I was back at square one - still had the F06 error...

Just like a few of you speculated, I decided to take a closer look at all the relays after I replaced them.  I tested each one individually during different operation modes of the UPS - (battery power only, mains power only, and battery+mains right before F06 fault takes place).  I found that only one relay was not switching at all - Relay RY14.  So I started looking more closely at this relay.  There's a digital NPN transistor (PDTD113ZT) and a 31V zener connected to the coil.  The base of the transistor connects to the microcontroller at a GPIO pin.  (I attached a schematic I drew of this portion of the circuit below).  Now that I understood the circuit, I decided to see what would happen if I injected ~1VDC at the base of the transistor, hoping it would switch on the relay.  I did this right before the F06 error would normally happen, and it switched the relay on... No fault occurred!! So it is indeed this RY14 not switching that is the cause of the issue!!

Now I measured the output of the microcontroller GPIO this transistor base connects to.  At no point in time did I measure a voltage > 0VDC.  So the micro is not sending a signal to switch on the relay.  Does anybody have any thoughts why? Could it be anything else besides the micro being faulty?  Thoughts for further testing/troubleshooting?? I greatly appreciate everyone's help so far.  Look forward to hearing your thoughts!! Here's a video I did explaining where I am at with this thing and the test I did.  It's a bit long, but it might help better understand the issue.  Thanks everyone!!