Repair Help for APC UPS BR1500G - F06 error (FIXED!!)

[Vovk_Z]

That's good you don't stop and now even can repair your own damage
I'm strugling to write "we told you to look into that output circuit carefully earlier".
It could be dangerous to microprocessor output pin when you apply external voltage to it without current limitting (like 100 R resistor in series). I would better cut a control wire to do it safe for microprocessor. Or I would just check a control voltage if it present or not.
Are you sure that relay RY14 has to move at your tests?

[RLP]

OK that is fascinating indeed. Great diagnosis!

My general feeling is that that particular GPIO on the microcontroller is probably dead - I have seen it happen. Usually it happens where a pin is exposed to some transient, so it's hard to guess why it would happen in this case. But definitely in the realm of plausibility. The fact that it thinks the relay is welded seems like proof that it 'should' be outputting a signal on that GPIO pin to control it. Although... I suppose there is still a tiny possibility that something else is wrong (e.g. voltage measurements) that is causing it to not switch when it actually should. But I think the voltage measurements must be correct: you said it outputs 120V (hence output measurement is probably correct) and the display shows input voltage correctly. Can the display show output voltage as well, to verify that...?

Replacing the microcontroller should be feasible if you can solder it, because you can almost certainly dump the firmware from the existing microcontroller. The STs are relatively simple to do this on: it's documented by ST, but basically something along the lines of setting the boot mode with BOOT0 and BOOT1 pins so that it runs the internal factory bootloader, and then powering it up and connecting to UART0 with something like a USB to serial adapter, and using ST's flash loader program to dump the firmware. Same process for programming a new chip. And they are usually unlocked on APC products so no read protection to stop you doing this!

[fmashockie]

That's good you don't stop and now even can repair your own damage
I'm strugling to write "we told you to look into that output circuit carefully earlier".
It could be dangerous to microprocessor output pin when you apply external voltage to it without current limitting (like 100 R resistor in series). I would better cut a control wire to do it safe for microprocessor. Or I would just check a control voltage if it present or not.
Are you sure that relay RY14 has to move at your tests?

I know Vovk  ! You and RLP were definitely right on the money from the start! Thank you for that tip about current limiting when applying external voltage to micro.  I really am in new territory here.  My experience troubleshooting micros is a bit slim.  I did limit the current from my power supply.  And I tested the micro prior to running this experiment to see if any voltage was being applied to signal RY14 - it was 0V throughout the start-up process.  Having trouble understanding your last question.  But I greatly appreciate your help!

Can the display show output voltage as well, to verify that...?

Yep the display can show input and output voltage. And from what I measured, both of them are very accurate. 

Replacing the microcontroller should be feasible if you can solder it, because you can almost certainly dump the firmware from the existing microcontroller. The STs are relatively simple to do this on: it's documented by ST, but basically something along the lines of setting the boot mode with BOOT0 and BOOT1 pins so that it runs the internal factory bootloader, and then powering it up and connecting to UART0 with something like a USB to serial adapter, and using ST's flash loader program to dump the firmware. Same process for programming a new chip. And they are usually unlocked on APC products so no read protection to stop you doing this!

So I'm not worried about desoldering this micro and soldering a new one.  I'm confident in my abilities to do that successfully (I like to use ChipQuik removal alloy.  Works really well and don't have to worry about applying too much heat like with hotair).  But reprogramming the chip and extracting the firmware is definitely new territory for me.  I've always wanted to learn how.  Especially because I've run into other repairs where having this experience would have allowed me to complete them successfully!! Do you have any resources/procedures you can share to get started? Do I need a USB programmer to do this (I'd need to purchase one if so).  Thank you so much RLP for everything so far!!

[fmashockie]

Well guys I'm pretty excited to give 'flashing' this STM32F1xx micro a try!  While I am still entertaining the possibility the micro is fine and there's something else wrong with the board, I'm leaning more towards an issue with the micro itself.  So I've done some research on how to flash this chip.  RLP you were pretty much spot on about how to flash the chip as far as I can tell.  ST has a ton of great resources for how to do this.  I started with reading ST Application Note 2606 (https://www.st.com/resource/en/application_note/an2606-stm32-microcontroller-system-memory-boot-mode-stmicroelectronics.pdf). 

The STM32F1 appears to be very easy to work with.  It is compatible with UART connection.  So I then bought one of these FTDI USB 2.0 to UART cable to interface with the micro (data sheet here https://ftdichip.com/wp-content/uploads/2020/08/DS_C232HD_UART_CABLE.pdf

Next it seems the ST also offers free software to download that will allow me to enable the bootloader function and flash the chip.  It's called the STM32CubeMx (info here https://www.st.com/en/development-tools/stm32cubemx.html).

I will also order a replacement micro as well to perform the flash (man this repair cost for this is starting to add up  )

But anyway, based on all that, does it sound like I am on the right track?? Like I said, this is all really new to me so if someone could check my thought process here, I would greatly appreciate it!!! Really excited to give this a try!!

[RLP]

Yeah that sounds good to me. I've only used the older software myself (https://www.st.com/en/development-tools/flasher-stm32.html) but I assume the new Cube software will have all the same functionality in there somewhere. Good luck!

[fmashockie]

Well guys I have another update with this UPS... It's fixed!!!    Well not exactly...

So I am waiting for the delivery of the UART-USB adapter and the new micro so I can flash the chip.  But today I decided to test the AVR function of the unit a bit more.  A few things I found out:

RY14 = Boost relay (this is the relay I discovered that was always off and was not getting a high input to the base of its controller transistor Q63)
RY12 = Buck relay (more about that later)

Boost mode = RY14 is on, RY12 is off
Normal mode = RY14 on, RY12 on
Buck mode = RY14 is off, RY12 on

So I decided to test the unit in boost mode.  I started bringing the input voltage down from 120VAC while injecting ~1V at the base of transistor Q63.  Once I got down to ~100VAC, the unit was in boost mode and it automatically triggered the F06 error.  So this meant there's something wrong with the buck relay (RY12) as well - it was not turning off once the voltage got low enough for boost mode!  The transistor that controls switching to RY12 is Q57.  I traced it back to the micro and it is connected to pin 27 which is another GPIO pin (PB14).  This is right next to the pin that ties to the base of Q63 for the boost relay RY14 (pin 27 PB13).  I have attached a layout of the micro and descriptions of pin functions. 

What I also discovered is that pin 27 is shorted to the VDD pin! I checked to see if there's anywhere else that this short could be occurring but could not find it (will continue looking).  Please correct me if I am wrong, but none of the GPIO pins should be shorted to a VDD pin right?  I was thinking this should be another indication that the micro is bad (maybe?).  But I decided to do a bit more experimentation and so I shorted the bases of Q63 and Q57 together with a 0 ohm resistor to see if the UPS would work under normal voltage (120VAC) conditions and it does!  So for that reason, it is fixed somewhat right?? 

Curious to hear if anyone has thoughts on this and/or if this new info changes anyone's ideas on the micro being bad?? Is there some other part of AVR circuitry I should check/test?  Regardless, I'm really excited to try to flash the micro once I get the stuff I ordered.  Here's a video below I did on these findings.  And as always, really appreciate everyone's help so far!!!

[BradC]

If I didn't know any better (and I don't!) I'd say it almost looks like the micro thinks it's on a board that doesn't have AVR.

I have no experience with later units than the late 4th Generation SUA units, but on all the APC units I've worked on, a firmware setting held the device type. I had once instance where the eeprom on a unit ended up corrupt and a 230V 50Hz unit believed it was a 110V 60Hz unit. All sorts of weird things happened.

I find it hard to believe you have a fully functioning ups *except* for hardware failures on both boost/buck relays simultaneously. One duff output perhaps, but for both to go with the rest of the unit performing normally sounds pretty remote to me.

[RLP]

I don't know much about the internals of microcontrollers or their failure modes - but a GPIO being shorted with low impedance to a supply rail while the MCU is off, does sound like evidence of an internal problem. The two pins in question are next to each other and on the same port (PBx), so I guess there is some possibility of a common cause for the failures.

Random thought: relay coils produce voltage spikes, which are usually clamped by diodes in the relay drive circuitry. Have you tested the diodes on those two relays? My thinking is: flyback diode fails -> back EMF from relay switching not clamped -> propagates through circuit and does damage to the MCU. It'd probably kill the relay drive transistors first though... did you mention that one may have been faulty?

I did have a microcontroller once where the entire ADC section failed, so all of the ADC inputs would read high values (as if the internal voltage reference was low), unless at least one ADC input was connected to a low-impedance voltage source such as VDD, then the remaining ADC inputs would read correctly. The rest of the microcontroller was fine, so reading firmware/replacing the chip/flashing fixed the problem. I never figured out what caused the problem, my only conclusion was that weird stuff can happen inside microcontrollers!

[fmashockie]

I don't know much about the internals of microcontrollers or their failure modes - but a GPIO being shorted with low impedance to a supply rail while the MCU is off, does sound like evidence of an internal problem.

Yep this is exactly what I am witnessing (should have been more clear).  This is with the mirco off and power disconnected from the board.  Still measuring short between PB14 pin and VDD pin.  That's gotta be a hardware failure of some kind right (whether on the micro or elsewhere)?

Random thought: relay coils produce voltage spikes, which are usually clamped by diodes in the relay drive circuitry. Have you tested the diodes on those two relays? My thinking is: flyback diode fails -> back EMF from relay switching not clamped -> propagates through circuit and does damage to the MCU. It'd probably kill the relay drive transistors first though... did you mention that one may have been faulty?

Interesting thought.  So I did replace the internally biased BJT (Q57) and it's zener diode which connect to RY12.  This was prior to having a better understanding of the circuit.  I just thought they were measuring a bit lower forward bias voltage during diode test - turns out it was normal.  But I will replace Q63 and D68 for RY14 just in case.  I know it won't fix the problem, but just to prevent any damage to the new micro.  Thanks for your continued help on this RLP - I greatly appreciate it!!!

If I didn't know any better (and I don't!) I'd say it almost looks like the micro thinks it's on a board that doesn't have AVR.

I have no experience with later units than the late 4th Generation SUA units, but on all the APC units I've worked on, a firmware setting held the device type. I had once instance where the eeprom on a unit ended up corrupt and a 230V 50Hz unit believed it was a 110V 60Hz unit. All sorts of weird things happened.

I find it hard to believe you have a fully functioning ups *except* for hardware failures on both boost/buck relays simultaneously. One duff output perhaps, but for both to go with the rest of the unit performing normally sounds pretty remote to me.

I hear what you're saying BradC, but I don't see how this could possibly be a firmware issue.  I didn't do ANYTHING to this unit/board.  It was in service for ~3-4 years, then I took it out service briefly/put into storage, then I took it out of storage once to take a look internally to help troubleshoot another UPS, then put it back into storage for several months, and finally took it back out recently and it started with this fault!  Did not make any modifications to it or anything.  It always worked fine until now.  So I really doubt this is a firmware issue.  It has to be a hardware fault of some kind.  But I appreciate your input! 

[Vovk_Z]

It looks like a hardware issue but APC UPS are known to have software problems too, e.g. loosing calibration parameters or even totally loosing all settings (230/115 V and/or 50/60 Hz etc). I've seen it on one of my UPS when smal electrolytes became a bit 'weak' (charging voltage became 24VDC instead of 27.6 VDC).

[fmashockie]

It looks like a hardware issue but APC UPS are known to have software problems too, e.g. loosing calibration parameters or even totally loosing all settings (230/115 V and/or 50/60 Hz etc). I've seen it on one of my UPS when smal electrolytes became a bit 'weak' (charging voltage became 24VDC instead of 27.6 VDC).

Wow that is fascinating Vvok! Yea I'm pretty preplexed as to what could be causing this.  I want to think its a problem with the micro due to that short of the PB14 pin to VDD.  I did a thorough check of the board tracing all traces and couldn't find anywhere else this short could be occurring. 

But at the same time, it almost seems too perfect of a fault to be a hardware issue.  Kind of like BradC says, everything works fine except the AVR. 

In your opinion Vvok, would flashing the chip and replacing the micro resolve this issue if it is a software problem? Is the programming corrupt on the original chip to prevent this from being successful?

I guess I will know soon enough.  Should be receiving the necessary items to do the flash today/tomorrow!!

[Vovk_Z]

IDK, try everything.

[fmashockie]

IDK, try everything.

Will do! I already got the replacement micro in.  Just waiting on the UART to USB cable which I should have by Monday.  I downloaded the STMCubeProgrammer software as well.  Seems really easy to use.  I'll give an update on how the flashing of the original micro goes, and the replacement with new micro.

If all that doesn't fix the issue, it would probably work just fine keeping the bases of Q63 and Q57 tied together with the original micro's PB14 and VDD pins shorted.  It just wouldn't have working AVR.  Who needs that anyway?   

[BradC]

It just wouldn't have working AVR.  Who needs that anyway?   

So if you get a mains spike or dip the first thing it'll do is dump your load with a relay weld fault. Makes the UPS more of a PS.

[fmashockie]

It just wouldn't have working AVR.  Who needs that anyway?   

So if you get a mains spike or dip the first thing it'll do is dump your load with a relay weld fault. Makes the UPS more of a PS.

Yes I know thats why it was a joke.  Although, where I'm at, mains is fairly stable.  We use these more for complete power failures.  It switches from mains to battery just fine with the buck and boost relays stuck in the on position.  I'd be willing to bet it would work just fine like this if I put it back into service.  But, I'd rather fix it for real.

[fmashockie]

Hey everyone - another update regarding this APC UPS BR1500G repair.  And unsurprisingly, it continues to be a stubborn bastard! 

I finally got in the items I needed to perform the flash of the micro - a FTDI UART to USB cable (datasheet attached), a replacement micro (it is a STMF102C8, datasheet attached) and I installed the STMCubeProgrammer software.  I installed the drivers for the FTDI cable as well from their website.  I also got a LQFP48 adapter board in case I had to remove the chip from circuit to flash it. 

I read the ST application note 2606 to get info on the bootloader function for this chip.  Here's how I set it up:
-Connected the VCC 3.3V (red) wire from the FTDI cable to VDD on micro
-Connected ground (black) wire from FTDI cable to VSS on micro
-tied BOOT0 (pin 44) on micro high (3.3V)
-tied BOOT1 (pin 20) on micro low (gnd)
-Connected the TXD (orange) wire from FTDI cable to PA9/USART1_TX (pin 30) on micro
-Connected the RXD (yellow) wire from FTDI calbe to PA10/USART1_RX (pin 31) on micro
-I selected 'UART' in STMCube software and checked to ensure serial configs matched (see photo attached).

Of course, once I was finished connecting everything, I tried to connect and an error message popped up saying 'Error activating device - check bootloader and serial configs and reset'.  I double checked everything and powered cycled my setup and tried again - same result.  I tried tying the NRST pin low to see if that would reset it, but that still gave me the same error. I tried connecting to the micro both in-circuit and out of circuit (note - I did verify that pin 27 is indeed shorted to VDD like I mentioned before!). 

So now I've reached another brick wall again!  I was hoping you guys could take a look at my set-up to make sure I'm not missing anything (I've attached photos as well).  Any help would be greatly appreciated!  Also, if the micro is read-protected by the manufacturer, would this explain the error message I am getting?

Thanks in advance as always!!

[RLP]

Sounds like you're close!

I remember being able to connect to the device and then view status of each flash region (protected/unprotected), so I imagine you should be able to connect even if it is read protected. Unless there is some separate protection setting that disables the bootloader entirely... that would be documented somewhere if it exists.

Do TX and RX need to be swapped? I might be missing something but I'd usually connect TX of the adapter to RX of the chip and vice versa.

Also my quick skim of AN2606 suggests the bootloader will auto-detect the baud rate - can you try lower baud rates than 115200?

[fmashockie]

Once again RLP you were spot on!

So I did indeed have the TXD and RXD wires reversed.  I also lowered the baud rate (115200 is the max according to application notes) and set parity to even.  After I fixed all that, I was able to get the original micro connected to the STMCubeProgrammer software no problem!  It appeared to read all the flash memory successfully.  So I extracted the flash memory and saved it to my PC.  I then desoldered the original chip from the LQFP48 adapter board and soldered in a brand new micro.  I made all the connections, configured the bootloader, and connected to the chip.  I downloaded the flash memory I saved from the original chip onto the new micro.  I verified the files were exactly the same.  Then I desoldered the new chip from the adapter board and soldered it onto the UPS PCB.  I powered it up and..... nothing happened... 

I verified the display was getting power.  It definitely appears that the new micro was not programmed properly.  I then connected the new micro to the STMCubeProgrammer again (thank god it worked in-circuit because I did not feel like desoldering it again!) to verify the flash memory was there and it is. 

Is there something else that I am missing as far as programming the chip?  Some crucial step perhaps? The only thing that seems out of the ordinary (maybe?) is that a few of the address lines in the flash memory have all zeros (see photo attached).  But that is how the flash memory from the original micro read as well.  Sorry, if I am not giving enough info - this is my first time attempting this.  I'll post in the microcontroller forum as well.  But any help you can provide would be greatly appreciated! I can't thank everyone enough so far!! Also, here's a video below summarizing where I'm at so far.  Thanks again!!

[darkspr1te]

I replied in the other thread but will post here too, you didnt download the full firmware so you need to reattach the original chip to the dev board and dump it,
 then check j40 for PA13/PA14 pins (pins 34/37 of mcu) as J40 looks like a programming header and you should be able to flash in circuit
when you dump original firmware set the read size to 0xffffff and it should read the whole chip.






darkspr1te

[DavidAlfa]

Better don't read 0xffffff, might crash or cause other issues later, I remember having issues in ST-Link utility, getting frozen/unresponsive after getting a bit over the limit.
Do the correct thing instead, we all know C8/CB parts have 128KB, so 0x20000 is the size.

[fmashockie]

Thank you both for your quick responses! So I guess when you say read siz, I should change size (see photo) from 0x400 to 0x200000 (or 0xffffff as darkspr1te said)??

Sorry this is all completely new to me so thank you bearing with me!

[DavidAlfa]

Exactly. Do that when reading it.
About VDD and PB14 shorted together... double check everything connected to it, suggests some external voltage spike fried the esd diode!

[fmashockie]

Thank you so much DavidAlfa! I will give it another go as soon as I get in to the lab today.  I let y'all know how I make out!

And I did verify that the VDD to PB14 short is on the micro.  That being said, doesn't rule out there's some issue on the board I haven't found that could cause it to happen again 

Will report back later!

[fmashockie]

Well everyone I am pleased to say that this APC UPS BR1500G is fixed! It put up quite a fight! 

So changing the read size to 0x20000 allowed me to flash the micro correctly! Need to learn to read hexadecimal lol.

Can't thank you guys enough for all the help.  Could not have fixed it without everyone's input!

Final video to the series for those interested in watching 

[RLP]

Awesome, glad to see you fixed it in the end!! A dead GPIO pin huh - who would've thought of that when presented with an "AVR relay weld fault"!

Repairs certainly are a great way to learn: IMO there is no better way to learn about electronics than to solve real problems like this, and see first-hand all the weird stuff that actually happens in circuits. Plus, who doesn't love a good mystery!