Eaton Powerware 9125 UPS (FIXED, very unusual fault!)

[TheMG]

I have a few of these UPS failed over the years, all pretty much the same way. Power goes out, long enough to deplete the batteries, UPS shuts down, and refuses to turn back on ever again (yes, even with good fully charged batteries). On some of them no sign of life at all, on others when you press (and hold) the power button the fans come on and the "bypass" LED lights, but nothing else happens.

First and foremost, these were out of warranty, let alone any form of manufacturer support whatsoever, a long time ago. I've scoured the internet and have found absolutely nothing for schematics or any service information of any kind. Seems Eaton has succeeded in keeping this information a well guarded secret.

These UPS have a socketed daughter board that contains all the analog and digital control logic, so I swapped in a board from a known good UPS into the failed ones and sure enough, they all fire up and work normally.

So in all cases, the fault is on the control board and not on the power (main) board.

The control board has at its heart a TMS320F243PGEA digital signal processor, an Altera Max EPM7128STC100-15 CPLD (glue logic and perhaps has a role in PWM pulse train generation), some gate driver ICs, couple of ULN2003 drivers, and some op-amps for signal scaling/conditioning. Also a AM29F200BB EEPROM and 25080 SPI EEPROM.

So far, I've checked voltages, +5V nice and solid, no problem there. Checked that the TMS320's crystal oscillator clock is working, and it is (while the power button is being pressed down). No IC is getting abnormally warm, and I've scoped the output of all opamps didn't find anything much out of the ordinary there compared to a known good working board.

I'm really beginning to suspect a fault with the TMS320 DSP, or firmware corruption. For one, the behavior of the indicator LEDs does not match what is supposed to happen, the "bypass" LED is never supposed to come on all by itself without an accompanying fault indication, and buzzer going off constantly (buzzer makes no sound at all right now). Also even though the bypass LED comes on, power is never provided to the output at all.

The good news is... the board has a programming header for the DSP and I've mapped out all of the relevant JTAG pins. Bad news, I don't have a JTAG interface and wouldn't have a clue how to use it! Anyone familiar with the TMS320 DSPs able to lend a hand in how to diagnose one, would be much appreciated!

[TheMG]

Well either my TL866 is acting up or the AM29F200BB EEPROM IC is messed up. I desoldered it off the PCB and put it in the programmer, when I do a read  it's as if the EEPROM is completely blank (FFFFFF throughout).

[TheSteve]

Double check the pins/socket very carefully - TSOP's can be pretty picky.

[TheMG]

I did a bunch of times. Then I pulled the EEPROM from a working PCB and dumped the contents which worked without a problem. Back to the suspect EEPROM, blank.

So I programmed the suspect device with the code dumped from the working board, it programmed and verified no problem. Soldered it back in, no change in the behavior of the defective board.

I wonder if maybe the TMS320 is actually faulty and not running the program correctly, resulting in deletion of the EEPROM contents?

Really weird problem.

[TheMG]

Actually I had a cold solder joint on one of the pins when I resoldered the EEPROM.

The board seems to live now! LEDs flash a few times and the buzzer comes on, then it shows a general failure indication (all LEDs on) as would be expected since I'm powering the board outside the UPS from a bench supply.

[TheMG]

It's ALIVE!!!!

Put the board back in the UPS and it does everything it's supposed to.

Honestly, this is the LAST thing I would have expected to be faulty in a UPS. How the heck did the EEPROM totally lose its memory? The datasheet states data retention should be good for at least 20 years, and the UPS is about 15 years old.

[TheSteve]

A very odd failure indeed. There must be some odd mode where it decides it needs to erase the flash.
Would you consider attaching the firmware for down the road when someone else has the same problem?

I am pretty sure my UPS at home is a Powerware 9125...

[perieanuo]

i see 3 scenarios, planned obsolescence triggered by some conditions, bad programming (that's my bet) or x-ray generator or something alike 'next' to your eep (saw this in medical equipment, eep died inside several microcontrolers, 'next' can mean 4-5 meters)

[TheMG]

I've got patient #2 on the workbench, a Powerware 9125 1000VA model this time, exhibiting similar symptoms. Unlike the other one, the flash EEPROM is not empty and seems to hold valid code. I of course can't confirm that it's not corrupt, but at least there's something there and not a blank EEPROM. I'll have to see if I can scrounge up another UPS of the same model in working condition and compare the EEPROM contents.

Did notice a TL084AC opamp on the board getting rather warm, which is unusual, replaced that but still no dice.

This one will require more in-depth troubleshooting. Would be a lot easier with at least access to schematics, but unfortunately none exist.

A very odd failure indeed. There must be some odd mode where it decides it needs to erase the flash.
Would you consider attaching the firmware for down the road when someone else has the same problem?

Attached. This is from a Powerware 9125 120V 2000VA model. I don't know if this firmware will work across different sizes/voltages. The small SPI EEPROM seems to hold model and configuration information but I can't be sure that there aren't model specific differences in the main firmware. Also, there seems to be at least 2 different hardware revisions of these control boards, the newer revision uses TMS320LF2407APGEA and is likely incompatible.

[NiHaoMike]

How the heck did the EEPROM totally lose its memory? The datasheet states data retention should be good for at least 20 years, and the UPS is about 15 years old.

That figure is at room temperature, if it's on a hot board it could be quite a bit less. Or it might be a defective one that didn't live up to specification. Read disturb could happen but unlikely since the description of the problem indicates it's probably only read at boot.

[TheMG]

I have a problem with that equipment, does anyone know where I can download the manual?

If you mean the user manual, it's readily available on the Eaton website.

If you're looking for a service manual... good luck, I haven't been able to find anything (for this and most other Eaton products).