Worn out flash memory chip (update: maybe not)?

[TheMG]

Have this network management card for a UPS which has been having some issues. The initial problem was it would randomly reset all the time, every few minutes. I attempted a defaults reset to see if that would alleviate the problem, but upon trying to reconfigure the card, every time I try to change the IP address, it crashes and when it comes back up it's back to the default IP.

So the next step I took was to operate the card on the bench from an external power supply, and check the onboard regulators (3.3V) for proper voltage and ripple. Everything looks ok there. While doing this, I noticed the card was not randomly resetting like it was doing when installed inside the UPS.

Thinking maybe the firmware is corrupt somehow, I attempted to reload the firmware... no luck, hangs in the middle of the firmware upload process.

My hypothesis is that the flash memory chip is simply worn out from too many write cycles, and any time it tries to write to a bad memory block it causes the card to crash and reboot.

The flash chip is a 39VF800A in a 48-TSOP package.

In order to replace the memory chip, I would need a reader/programmer to make a copy of the contents of the memory, and write the contents to the new memory before soldering it to the board. I've been putting off getting a programmer for a while and this is as good an excuse as any to finally buy one. Can anyone suggest a decent universal chip programmer capable of handling this chip (as well as other common memory chips and micro-controllers)?

Also as a side note, can anyone identify the CPU on this thing? AC1101 brings up absolutely nothing in any datasheet searches. I can't even identify the manufacturer, as I don't recognize the symbol.

[sokoloff]

For $17 to $25, you can buy another one.

Given that, I wouldn't spend any time on it.

https://www.ebay.com/sch/i.html?_nkw=ConnectUPS-X&_sacat=0&_sop=15

[TheMG]

I realize that. However, I'm in Canada and most of those for sale are in the USA. With the exchange rate, shipping fees, etc, that's $90+ by the time it's all said and done.

That, and what's to say it's not going to soon suffer the same fate, since most of these for sale have probably been pulled out of units that were in service for many years, and thus would have significant write cycles on the flash already.

Anyways, I will need a programmer for some upcoming projects anyways so I wouldn't be buying it strictly for this one repair. Plus I like the challenge of trying to repair things if I can, rather than throwing away.

[picburner]

The TL866II eprom programmer supports this type of memory,  you will also need its TSOP48 - DIP40 adapter of course.

[TheSteve]

I would get a TL866II Plus with the various adapters - they sell it all as a single kit.
Although I am more suspect of ram then flash.

Unless you're near Langley BC and then I'm happy to read the old one and write a new one for you.

[TheMG]

What makes you think that it writes to the flash memory in normal operation?  It appears to have battery backed RAM.

The battery is only for the RTC. Logs and configuration are retained even when the battery is removed for an extended period of time. There is an RTC chip on the main board and it's the only thing the battery feeds power to.

I would get a TL866II Plus with the various adapters - they sell it all as a single kit.

$68 shipped from China, wow! Almost seems too good to be true, but the reviews seem to be pretty good. Purchased one along with the appropriate adapters. Will be useful for programming PIC micros in the future too, instead of buying more expensive ones with a pre-loaded bootloader. Also have a few PICs where I've "bricked" the bootloader so it'll be useful for that too.

Unless you're near Langley BC and then I'm happy to read the old one and write a new one for you.

Yellowknife...

[negativ3]

In Canada...

https://www.ebay.com/itm/POWERWARE-CONNECTUPS-X-WEB-SNMP-xHUB-CARD-5071-00010-A5-price-reduced/183710118030?epid=1869348947&hash=item2ac5f9f08e:g:amYAAOSwcyVceWzs

[TheMG]

Well I got the chip programmer, removed the flash chip from the PCB, and read its contents.

It doesn't look like the flash memory is to blame, after all, as I've written all FF and all 00 to it and that seems to be fine.

But the fact remains, the device is still bricked from the failed firmware upload with no way of recovery besides trying to flash the memory directly.

Interestingly, comparing the firmware binary obtained from the manufacturer's website with the contents of the flash memory, it almost looks like the firmware flashing process started to write the firmware near the bottom of the memory, and then "wrapped around" back to the top and kept going from there. Now, maybe this is by design, but seems odd to me especially since the first part of the firmware binary appears to be some type of BIOS. Why would BIOS code be so far down in the memory and not at the very top? Seems suspect.

Interestingly, there is in this BIOS what seems to be some diagnostic functions for testing flash, RAM, etc. I could never figure out how to access these "hidden" functions.

So there's a couple of things I need to figure out:

1 - How to correctly write the firmware to the flash memory. Tempted to try just write the binary firmware file to it sequentially starting at location 00000000 and see what happens.

2 - Test the RAM chips somehow, or just replace them.

I may be in way over my head on this one, as I'm definitely not much of a programmer, especially since this is a completely unknown CPU with no documentation to be found anywhere.

Oh, and, for the record, I did find another one of these cards (newer hardware revision actually) new-in-box for a decent price. But I'd still like to attempt to fix this other one, if I can, if for nothing other than the experience.

Binaries attached. The one named "PW338.bin" is the firmware flash file as available on manufacturer's website, the other one is what I've extracted from the actual flash chip.

[fzabkar]

AFAICT the firmware consists of 4 modules:

PW338.bin

Code: [Select]

offset   size   description
------  -----   ---------------------------------------------------------
    0     200   0x200 byte header for firmware download (checksum = 0x00)  
  200    f000   BOOT Generic - 0x100 byte header + code
 f200    1000   BOOT Generic - 0x100 byte header + boot code
10200   4b3d0   FILE Ingrasys USHA - 0x100 byte header + code
5b5d0   4f800   MAIN ConnectUPS - 0x100 byte header + code
aadcf           end of firmware payload

flash_contents.bin

Code: [Select]

offset   size   description
------  -----   ---------------------------------------------------------
    0   a0000   FILE Ingrasys USHA - 0x100 byte header + code
a0000   50000   MAIN ConnectUPS - 0x100 byte header + code
f0000    f000   BOOT Generic - 0x100 byte header + code
ff000    1000   BOOT Generic - 0x100 byte header + boot code
fffff           end of flash

The firmware payload file (PW338.bin) has a 0x200-byte header which appears to define the destination location and size of each component, but I haven't been able to determine how this is done. This would explain why the modules appear to be out of order. There is a single checksum byte at the end of the header (offset 0x1FF, value 0xB8). This byte is chosen so that the sum of all 0x200 bytes in the header is 0x00.

The BOOT block appears to be in the right place for an Intel x86 processor architecture. An Intel CPU begins executing at FFFF:0.

Code: [Select]

Offset(h) 00       04       08       0C

000FFFF0  EA000100 FFFFFFFF 30362F32 322F3032  ........06/22/02

If you disassemble the code at this point, it executes a JMP instruction which sends it to the first instruction after the 0x100-byte header of the "BOOT Generic" block (FF00:0100).

Code: [Select]

C:\>debug
-e 100
1262:0100  EA.ea   00.00   01.01   42.00   A0.ff
-u 100
1262:0100 EA000100FF    JMP     FF00:0100

This JMP behaviour is consistent with PC BIOS.

If we now extract each of the 4 components from the flash.bin dump, using the size in the payload, we find that the firmware update aborted part way through the final module, "MAIN ConnectUPS", at offset 0xE8000 in the flash. All other modules match up (apart from a mysterious 4-byte parameter). ISTM that you just need to patch the flash by adding the rest of the "MAIN ConnectUPS" module at 0xE8000. I can do this for you if you can't understand what I'm suggesting.

The only query I have is in regard to the meaning of those mysterious 4 bytes in the headers of each of the 4 modules.

For example ...

PW338.bin

Code: [Select]

Offset(h) 00       04       08       0C

0005B5D0  5A3F0000 00C02603 00000600 00000A00  Z?..............
0005B5E0  08000110 FFFFFFFF 4D41494E 00000000  ........MAIN....
                   ^^^^^^^^
0005B5F0  436F6E6E 65637455 50530000 00000000  ConnectUPS......

flash_contents.bin

Code: [Select]

Offset(h) 00       04       08       0C

000A0000  5A3F0000 00C02603 00000600 00000A00  Z?..............
000A0010  08000110 2EC48C9E 4D41494E 00000000  ........MAIN....
                   ^^^^^^^^
000A0020  436F6E6E 65637455 50530000 00000000  ConnectUPS......

[amyk]

Great work, fzabkar! I decided to follow the jump over to FF100 and it looks like valid x86 code to me --- probably 8088/8086-level, since I didn't see any newer instructions. It starts off doing what looks like a self-test of the CPU, does some I/O, then goes to either FB300 where it copies some code into probably RAM and then goes to 1000:0200, or to F0100 with more CPU testing and I/O, maybe a memory test, and a lot more things happen.

Looking at the firmware image, I see what looks like .class filenames and reference to .jar --- which could mean this system runs Java

[fzabkar]

I have extracted the components from PW338.bin and flash_contents.bin:

http://www.users.on.net/~fzabkar/temp/UPS_firmware_components.7z

The PW338_MAIN.bin component has a 0x100 byte header, followed by a JMP instruction which jumps over a table of modules and lands on a LOADER module at A042:0100 (= 0xA0520). Note that this MAIN module has an absolute offset of 0xA0000, so the offsets in the following dumps should be viewed as being relative to this base address.

Code: [Select]

Offset(h) 00   02   04   06   08   0A   0C   0E

00000100  EA00 0142 A000 0000 0000 0000 0000 0000

Code: [Select]

-u 100
1262:0100 EA000142A0    JMP     A042:0100

Code: [Select]

Offset(h) 00   02   04   06   08   0A   0C   0E

00000520  EB0A 0100 4C4F 4144 4552 0000 6800 20E8  ....LOADER......

Here is the table:

Code: [Select]

Offset(h) 00   02   04   06   08   0A   0C   0E

00000130  53A1 0000 0000 0000 5061 7373 5468 7275  ........PassThru
          ^^^^                ^^^^^^^^^^^^^^^^^^^
absolute segment address      module name

00000140  5C0B 0000 2A0A 0000 0100 0000 0000 0000  ................

00000150  ECA2 0000 0000 0000 554C 5A00 0000 0000  ........ULZ.....
00000160  4516 7F00 7114 CC0D 0100 0100 0000 0000  ................
00000170  A2A3 0000 0000 0000 5354 5249 4E47 5800  ........STRINGX.
00000180  7D78 1330 F173 8B42 0200 0100 0000 0000  ................
00000190  7FA4 0000 0000 0000 5359 5354 454D 0000  ........SYSTEM..
000001A0  6528 BD58 B125 8619 0100 0100 0000 0000  ................
000001B0  A8A8 0000 0000 0000 5345 5256 4943 4500  ........SERVICE.
000001C0  EF06 6502 1E05 B803 0100 0100 0000 0000  ................
000001D0  41AA 0000 0000 0000 4D49 424D 4752 0000  ........MIBMGR..
000001E0  F349 AD08 5F3F 152D 0100 0900 0000 0000  ................
000001F0  7DAA 0000 0000 0000 454D 4400 0000 0000  ........EMD.....
00000200  7B15 9D05 7D12 4D0A 0100 0100 0000 0000  ................
00000210  4FAD 0000 0000 0000 5550 534D 4752 0000  ........UPSMGR..
00000220  B360 9D74 D355 D635 0100 0100 0000 0000  ................
00000230  F4AD 0000 0000 0000 5445 4C4E 4554 0000  ........TELNET..
00000240  7D59 D752 E656 DC39 0100 0100 0000 0000  ................
00000250  52B1 0000 0000 0000 5443 5049 5000 0000  ........TCPIP...
00000260  BB2B 7F28 F228 7B1A 0100 0900 0000 0000  ................
00000270  F0B4 0000 0000 0000 534E 4D50 0000 0000  ........SNMP....
00000280  ED3A 7F83 0B36 BA22 0100 0100 0000 0000  ................
00000290  98B6 0000 0000 0000 4854 5450 0000 0000  ........HTTP....
000002A0  9D24 B216 0523 7816 0100 0900 0000 0000  ................
000002B0  C4B8 0000 0000 0000 4448 4350 0000 0000  ........DHCP....
000002C0  9B2A 8129 7125 741B 0200 0100 0000 0000  ................
000002D0  2CBA 0000 0000 0000 5546 5450 5300 0000  ........UFTPS...
000002E0  D340 8311 543B 2423 0100 0900 0000 0000  ................
000002F0  E4BB 0000 0000 0000 534D 5450 0000 0000  ........SMTP....
00000300  8354 5124 7836 FD2D 0100 0900 0000 0000  ................
00000310  17BE 0000 0000 0000 504F 5033 0000 0000  ........POP3....
00000320  DFA2 FD41 5F8C 3E5A 0200 0100 0000 0000  ................
00000330  F7C0 0000 0000 0000 554E 4D50 5300 0000  ........UNMPS...
00000340  CD1A A507 9212 680D 0100 0900 0000 0000  ................
00000350  9BC6 0000 0000 0000 4E54 5000 0000 0000  ........NTP.....
00000360  B375 DB04 5C2C 8D37 0100 0100 0000 0000  ................
00000370  72C7 0000 0000 0000 4C4F 474D 4752 0000  ........LOGMGR..
00000380  9D3C 0500 E202 E40B 0100 0300 0000 0000  ................
00000390  EBCA 0000 0000 0000 4D49 4200 0000 0000  ........MIB.....
000003A0  7B3B 0000 8802 5E1A 0100 0300 0000 0000  ................
000003B0  AACB 0000 0000 0000 4D45 4E55 0000 0000  ........MENU....
000003C0  8921 AF1A 061D BE14 0100 0100 0000 0000  ................
000003D0  50CD 0000 0000 0000 7570 7354 7261 7000  ........upsTrap.
000003E0  BC8F D62C B85D BE28 0100 0900 0000 0000  ................
000003F0  9CCE 0000 0000 0000 4352 5950 544F 0000  ........CRYPTO..
00000400  1A96 623F 2F89 5A4B 0100 0900 0000 0000  ................
00000410  28D1 0000 0000 0000 5253 4100 0000 0000  ........RSA.....
00000420  B121 D52F 431A 9B15 0100 0900 0000 0000  ................
00000430  DED5 0000 0000 0000 5353 4800 0000 0000  ........SSH.....
00000440  45F5 C3A7 63C0 997C 0100 0900 0000 0000  ................
00000450  38D7 0000 0000 0000 4341 0000 0000 0000  ........CA......
00000460  812A 69AC 7628 2E18 0100 0900 0000 0000  ................
00000470  02DF 0000 0000 0000 4854 5450 0000 0000  ........HTTP....
00000480  070A 0901 B708 7D05 0100 0300 0000 0000  ................
00000490  85E0 0000 0000 0000 4445 5445 4354 0000  ........DETECT..
000004A0  F16F 5F2F 6564 4645 0100 0500 0000 0000  ................
000004B0  DDE0 0000 0000 0000 5550 4D00 0000 0000  ........UPM.....
000004C0  6359 0B11 3649 0833 0100 0500 0000 0000  ................
000004D0  32E5 0000 0000 0000 5550 4D00 0000 0000  ........UPM.....
000004E0  55CC F12E 4C95 3A6F 0100 0100 0000 0000  ................
000004F0  63E8 0000 0000 0000 5550 5344 4400 0000  ........UPSDD...

Each module in the table is defined by a 32-byte record. This record includes the absolute segment address and the module name.

For example, the "PassThru" module is located at absolute address 0xA153:0 (0xA1530). This is consistent with its location in the flash, which confirms that the firmware is being updated at the correct addresses.

This is the header of the payload file:

Code: [Select]

Offset(h) 00        04       08       0C       10       14       18       1C

00000000  5553 4841 01010002 03002000 08000110 31313134 32303132 2D19EE9F FFFFFFFF

00000020  4000 D8BD 80D00200 00000000 00000100 00000F00 00B00904 00000100 00000000  BOOT 1 & 2
00000040  6000 AE81 00D00200 00000100 D0B30400 00000000 00F00100 D0B30400 00000000  FILE
00000060  FFFF A73C 00D00200 D0B30500 00F80400 00000A00 00C02603 00F80400 00000000  MAIN
          ^^^^ ****          ^^^^^^^^ -------- ^^^^^^^^          --------
           |   CRC-16 CCITT     |      size    flash address       size
offset to next                  |
 record                      relative offset in payload                             
.......
000001E0  0000 0000 00000000 00000000 00000000 00000000 00000000 00000000 000000B8
                                                                       checksum ^^

It would appear that the two BOOT components have been lumped together.

I don't understand all the parameters for each component, but it is now clear how the updater knows where to find them and where to write them, and how to verify their integrity.

Clearly, all the data are little-endian.

The CRC-16 CCITT checksums can be confirmed with HxD (a freeware hex editor):

https://mh-nexus.de/en/hxd/

[TheMG]

Wow!!! I'm going to need a little bit of time to absorb all this, if I understand the overall picture, the data is in the right place, but one of the firmware "modules" is incomplete due to the firmware upload crapping out mid-process.

How did you find all this anyways? Either you have some magical software and/or some mad programming skills. Either way, I'm impressed. Much of the contents of the firmware might as well be a foreign language to me, granted my only programming experience thus far is with PicBasic, Arduino, and some assembler on the quite obsolete 68HC11s, so very limited in the grand scheme of things. Never touched anything x86.

So ultimately... it should be a matter of inserting the missing code into the overall binary file using a hex editor, write the resulting, hopefully complete and correct, binary to the flash chip, and in theory the card should be operable (at least in a limited capacity, assuming there's still a hardware fault on it of some sort which created the problem in the first place)?

[amyk]

How did you find all this anyways? Either you have some magical software and/or some mad programming skills. Either way, I'm impressed.

Reverse-engineering skills, not to be confused with programming... it usually starts with a few educated guesses and goes on from there.

[fzabkar]

So ultimately... it should be a matter of inserting the missing code into the overall binary file using a hex editor, write the resulting, hopefully complete and correct, binary to the flash chip, and in theory the card should be operable (at least in a limited capacity, assuming there's still a hardware fault on it of some sort which created the problem in the first place)?

Yes, that would be my approach. I would suggest that you let us see your final compilation before committing it to flash.

As for my methods, yours is not the first firmware which I have "reverse engineered", so it wasn't as difficult as it might otherwise have been. I'm not a programmer, but I can write simple programs. (I can also fry an egg, but that doesn't make me a chef.)

That said, the only skills you need for this particular task are a familiarity with hexadecimal arithmetic and an ability to recognise patterns in the data. But that's still only the easy part. Disassembling the code is much more challenging -- that's where the real reverse engineering knowledge is. I confess that I haven't done much in that regard.

As for tools, I used a freeware hex editor (HxD) for file viewing, carving, comparison and checksum calculation, and MSDOS DEBUG.EXE for disassembly.


Edit:

This is the repaired flash dump, AISI. I suggest you repair yours independently, just in case we don't agree.

http://www.users.on.net/~fzabkar/temp/UPS_flash_contents_repaired.7z

[TheMG]

Well, I patched the flash bin file to the best of my abilities, then used the compare feature in HxD to compare with yours, and it claims the two to be identical.

I guess the next thing to do now is to write that to the flash memory and solder it back onto the PCB and see what happens.

The stuff in the BOOT module at offset 000F306F in the flash still intrigues me:

Code: [Select]

1) CPU SELF TEST
2) SRAM MEMORY TEST
3) NIC INTERNAL TEST
4) FLASH ROM TEST
5) LED BLINKING TEST
6) READ DIP SWITCH STATUS
7) NIC EXTERNAL LOOPBACK TEST
9) REAL TIME CLOCK TEST
C) REAL TIME CLOCK ADJUST
F) FLASH FUNCTIONALITY TEST
A) I2C TEST

This would suggest some debugging functionality is built-in the BOOT module, but the million dollar question is... how can it be accessed/invoked?

[amyk]

This would suggest some debugging functionality is built-in the BOOT module, but the million dollar question is... how can it be accessed/invoked?

Load the file into a disassembler and do some tracing. You'll need to find how that code is reached.

There is, of course, always the possibility that it's completely "dead code" and unreachable, left over remnants from an actual test ROM.

[TheMG]

Well, after knocking over my bottle of rosin flux all over the work bench and having a huge sticky mess to clean up...

It's ALIVE!!!

What's more, it let me configure the IP address without crashing.

It is possible there's still a problem (faulty RAM maybe?) but this is a different firmware version from what was originally on the card so stuff relating to the IP address setting could simply be in a different, non-faulty, location in RAM.

So it's too early to tell whether it's fixed for good or not.

Next test is to chuck the card into one of my spare UPS and letting it run for a while to see if it really is stable or if it still has issues randomly crashing. For the time being, powering the card on the bench it seems good.

[fzabkar]

I just have one cautionary observation.

The headers of the BOOT, FILE and MAIN modules each have a 4-byte parameter which is set to 0xFFFF in the payload, but which changes when the module is written to flash. This would suggest that this parameter is unique and is in some way dependent on the target PCB. Perhaps it is a security related CRC??

In any case, I would be reluctant to write this flash dump to a different PCB, just in case it is indeed unique.

As for a Debug Monitor, what is the function of jumper block JP1?

[elie13]

Hello
thanks for the information you provided, i have the same problem with the connectups-X card, please can you share the latest modification firmware?, did you try after to flash it normally through serial port with the manufacture firmware? thanks for your help