Agilent DSOX3054A not booting possible NAND corruption HELP! (RESOLVED)

[analogRF]

after a couple of years (!) of searching and not being able to buy a 3000A/T model I finally got hold of a defective DSOX3054A which is stuck at boot. After a quick flash of several LEDs, only the REF and MATH LEDs remain on and nothing else.

Attached is what I got on the serial console and it looks to me to have a NAND flash corruption but I am not sure, I hope someone can confirm
I also ran a couple of informational commands after stopping the U-Boot which you can see in the attached file and below

Code: [Select]

U-Boot 2010.03 (Jan 26 2011 - 12:37:34)Agilent P500

CPU:   SPEAr600
DRAM:  128 MiB
Flash: 512 KiB
NAND:  128 MiB
In:    serial
Out:   serial
Err:   serial
SerNum:serial number not programmed
Chip:  BD Board Rev: 4
Net:   smsc
Press space to stop autoboot:  0
## Booting kernel from Legacy Image at f8050000 ...
   Image Name:   PBOOT
   Created:      2011-01-27  11:38:14 UTC
   Image Type:   ARM Linux Kernel Image (gzip compressed)
   Data Size:    36703 Bytes = 35.8 KiB
   Load Address: 00000000
   Entry Point:  00000000
   Uncompressing Kernel Image ... OK

Starting kernel ...


Debug serial initialized ........OK

Microsoft Windows CE Bootloader Common Library Version 1.4 Built Jan 27 2011 02:04:15
Microsoft Windows CE 6.0 Ethernet Bootloader for the Agilent P500 board
Adaptation performed by Agilent Technologies (c) 2008


System ready!
Preparing for download...
 Loading image 1 from memory at 0xD0600000
O
BL_IMAGE_TYPE_BIN

X
XXXXOOOOXOOOOOOOOXOXXOOOOOOOOXOOXOXOOOOXXXOOOOOOOOOXOOOOXOXXOXXOXOOOXOXXXXXXOOXXOOOOOOXOOXXOXXOOOERROR: Checksum failure (expected=0x1217115 

computed=0x1217126)
****** Checksum failure on record 36, ABORT!!! ******

Completed file(s):
-------------------------------------------------------------------------------
[0]: Address=0x80361000  Length=0x12F9870  Name="" Target=RAM
 Loading image 1 failed, trying next one
 Loading image 2 from memory at 0xD1600000

BL_IMAGE_TYPE_UNKNOWN

 Loading image 2 failed, trying next one
 All images failed

Press r to reset




================================================================================================

p500> imls
Legacy Image at F8000000:
   Image Name:   XLOADER
   Created:      2010-09-24  21:30:18 UTC
   Image Type:   ARM Linux Firmware (uncompressed)
   Data Size:    5370 Bytes = 5.2 KiB
   Load Address: d2800b00
   Entry Point:  d2800b00
   Verifying Checksum ... OK
Legacy Image at F8010000:
   Image Name:   UBOOT
   Created:      2011-01-26  19:52:00 UTC
   Image Type:   ARM Linux Kernel Image (uncompressed)
   Data Size:    194572 Bytes = 190 KiB
   Load Address: 03f00000
   Entry Point:  03f00000
   Verifying Checksum ... OK
Legacy Image at F8050000:
   Image Name:   PBOOT
   Created:      2011-01-27  11:38:14 UTC
   Image Type:   ARM Linux Kernel Image (gzip compressed)
   Data Size:    36703 Bytes = 35.8 KiB
   Load Address: 00000000
   Entry Point:  00000000
   Verifying Checksum ... OK
p500> nand
nand - NAND sub-system

Usage:
nand info - show available NAND devices
nand device [dev] - show or set current device
nand read - addr off|partition size
nand write - addr off|partition size
    read/write 'size' bytes starting at offset 'off'
    to/from memory address 'addr', skipping bad blocks.
nand erase [clean] [off size] - erase 'size' bytes from
    offset 'off' (entire device if not specified)
nand bad - show bad blocks
nand dump[.oob] off - dump page
nand scrub - really clean NAND erasing bad blocks (UNSAFE)
nand markbad off [...] - mark bad block(s) at offset (UNSAFE)
nand biterr off - make a bit error at offset (UNSAFE)
p500> nand info

Device 0: NAND 128MiB 3,3V 8-bit, sector size 128 KiB
p500> nand bad

Device 0 bad blocks:
p500>

I measured most or all of the voltages marked on board and they seem to be OK except one VP23V which is always 0V (see attached picture)
Other voltages are OK but some are less close than I thought they should be

The front USB has no power at all unless I stop the U-Boot then it gets 5.54V which seems a bit high!? (see the picture)

that's as far I have been able to go.

I have been/am reading the very very long thread on 2000/3000 and it is really next to impossible to find clear answer in 106 pages 

if someone can give me a direction it is highly appreciated.

[TheSteve]

Appears to be classic NAND corruption.

[nctnico]

How about sending it to Keysight for a free repair?

[analogRF]

How about sending it to Keysight for a free repair?

are they still doing that free replacement thing? this is quite an old scope, Agilent branded S/N MY5316xxx

[analogRF]

How about sending it to Keysight for a free repair?

I would also like to repair it myself as a good experience.
I dont have a lan card (yet) but is it possible to boot it by uploading the image through the YMODEM method (serial  console)?

is there any way to know what version of FW it has had?

[Pinkus]

My DSOX4K (one of the first productions) had the NAND problem too. I sent it to Keysight together with the printouts (UBoot etc.) to proof it would be the NAND. Also I mentioned in my letter that there is a service note which allows a free repair when NAND is the problem. It was repaired for free without any question. That was about a year ago.

[analogRF]

Based on Keysight website, this scope's warranty ended in Jan 2017. It seems the unit must have been manufactured in May 2013 and it has a Cal certificate with that date. No other history apparently...

Still I appreciate if someone can point me to how it can be recovered by LAN card and/or using the serial console.
I have talked to one of the forum members for a home made LAN card and hopefully I'll get that soon.
USB is powered when I stop the U-Boot, there should be a way to upload the image to through USB, no?

[TheSteve]

The new firmware is read in via USB but UBOOT has no support for it, the scope needs to be booted up. The firmware can be loaded via the serial console but that will not always work. Depending on the level of corruption the scope may not finish booting from the image enough to select the firmware upgrade option. I do all recoveries via network. Often I need to try various firmware versions to see which will boot as you need them to match, or be very close. Without booting and being able to telnet in I'm not aware of anyway to know the exact version installed.

[analogRF]

The new firmware is read in via USB but UBOOT has no support for it, the scope needs to be booted up. The firmware can be loaded via the serial console but that will not always work. Depending on the level of corruption the scope may not finish booting from the image enough to select the firmware upgrade option. I do all recoveries via network. Often I need to try various firmware versions to see which will boot as you need them to match, or be very close. Without booting and being able to telnet in I'm not aware of anyway to know the exact version installed.

I am waiting for a LAN card from one of the members.
In the mean time, can you please point me to the correct/verified procedure for the recovery from network?
Do I need to set up tftp on my laptop?

[analogRF]

Ok I just tried the YMODEM method using serial console with nk.nb0 extracted from 2.35 FW
I followed instructions by titiris here (msg # 2092)
https://www.eevblog.com/forum/testgear/dsox2000-and-3000-series-licence-have-anyone-tried-to-hack-that-scope/msg2136181/?topicseen#msg2136181

the uploading went well in 35 minutes but after executing the 'go' command this is what I got:

Code: [Select]

p500> loady 0x0361000 115200
## Ready for binary (ymodem) download to 0x00361000 at 115200 bps...
CCxyzModem - CRC mode, 1(SOH)/19482(STX)/0(CAN) packets, 5 retries
## Total Size      = 0x013064d4 = 19948756 Bytes

p500> go 0x00362000

## Starting application at 0x00362000 ...
Windows CE Kernel for ARM (Thumb Enabled) Built on Jan 24 2013 at 14:52:37
Setting up for a Cold Reboot
Done Setting up for a Cold Reboot
Windows CE Firmware Init
BSP 1.0.0 for the SPEARHEAD600AB board (built Jun 18 2013)
Adaptation performed by ADENEO (c) 2005
+OALIntrInit
-OALIntrInit(rc = 1)
Initialize driver globals Zeros area...
pDrvGlobalArea 0xa0060000  size 0x800 (0xa0060800 -0xa0060000)
Initialize driver globals Zeros area...done
 OALKitlStart
Firmware Init Done.
OALIoctlHalEnterI2cCriticalSection init i2c cs
ERROR: C:\WINCE600\PLATFORM\COMMON\SRC\SOC\STM\SPEARHEAD600\DRIVERS\GPIO\.\sh600_gpio_hw.cpp line 170: GPB driver, RegQueryDword('ISTPriority') failed, status:2
ERROR: C:\WINCE600\PLATFORM\COMMON\SRC\SOC\STM\SPEARHEAD600\DRIVERS\GPIO\.\sh600_gpio_hw.cpp line 170: GPB driver, RegQueryDword('ISTPriority') failed, status:2
++SER_Init: context Drivers\Active\14
SER_Init, dwIndex:2
SER2 got sysintr:0x00000017
SER2 Serial Port, new baud rate:0x1c200  (UARTCLK:48000000 IBRD:0x1a FBRD:0x2)
OHCI\system.c, GCFG_USBH1_SW_RST
OHCI\system.c, GCFG_USBH2_SW_RST
-EDeviceLoadEeprom 00:30:D3:20:DA:4C
Phy found addr 31 (ticks=2707)
WaitForLink Start (ticks=2709)
No Link (ticks=3711)
<--EDeviceInitialize

GMAC DMA status register = 0x0
BALDWIN_DDI: cBaldwinHwIf::Init: Initializing...
BALDWIN_DDI: cBaldwinHwIf::Init: Scope successfully identified.
BALDWIN_DDI: cBaldwinHwIf::Init: Success!
Running infiniiVisionInstallHelper
ERROR: OALIoCtlHalGetDeviceInfo: Device doesn't support IOCTL_HAL_GET_DEVICE_INFO::SPI_GETBOOTMENAME
Failed to start/configure network.
Time for NANDFLASH to load: 0 ms.
Time for SNANDFLASH to load: 0 ms.
Time for USB Hard Disk Drive to load: 0 ms.
Starting ProcessStartupFolder
ProcessStartupFolder: \usb\Startup\
                                   running \usb\Startup\infiniivision.lnk...

any clue?

is it because perhaps the version on the unit does not match 2.35?

[wraper]

Keysight repairs these scopes with NAND corruption for free even with expired warranty by replacing main board, as it's a widespread issue. Trying repairing this issue yourself is lame and might end with a very large expense, as it may end with you paying a full repair price to keysight. Not to say you will be left with old crappy NAND

[TheSteve]

The original corruption is likely preventing it from booting all of the way. It could also be a version mismatch.

[analogRF]

The original corruption is likely preventing it from booting all of the way. It could also be a version mismatch.

Can the network method work then?

is it possible that the dead 23V voltage (see the first post) is the reason?
so far i though maybe that voltage will come alive after certain point in boot process
but maybe it should always be there?

I wish someone could be kind enough to check when that VP23V line comes on after turn on

[TheSteve]

The original corruption is likely preventing it from booting all of the way. It could also be a version mismatch.

Can the network method work then?

is it possible that the dead 23V voltage (see the first post) is the reason?
so far i though maybe that voltage will come alive after certain point in boot process
but maybe it should always be there?

I wish someone could be kind enough to check when that VP23V line comes on after turn on

I am quite sure with a network card we'll get it to work. Send me a PM when you get it and we'll take care of it.

[Jwalling]

The document that Keysight did not make public for some reason.

[analogRF]

The document that Keysight did not make public for some reason.

Thanks, this can really be helpful if I want to send it to Keysight for repair

well well well...according to this document, they may not consider my scope as having a NAND corruption after all
I dont have a boot loop and the LEDs dont  "continually flash through the boot sequence"
what happens is that first a bunch of LEDs very quickly flash simultaneously and then MATH and REF LEDs come on and stay on forever.

If they really stick to their flowchart, then I guess they will consider this as a normal repair procedure = $4026 CAD + TAX 
which is a non starter for me

[analogRF]

even though there is no doubt that this is a NAND corruption issue but I am beginning to think that not having that 23V rail
could also be a secondary problem that prevented from booting after I uploaded the image through serial port

does anybody know if that 23V rail should always be ON after power is switched on or if it comes alive after a while?

[Pinkus]

well well well...according to this document, they may not consider my scope as having a NAND corruption after all
I dont have a boot loop and the LEDs dont  "continually flash through the boot sequence"
what happens is that first a bunch of LEDs very quickly flash simultaneously and then MATH and REF LEDs come on and stay on forever.

If they really stick to their flowchart, then I guess they will consider this as a normal repair procedure = $4026 CAD + TAX 
which is a non starter for me

a) As Keysight is replacing the main board when the NAND problem occurs, it seems just to re-install the firmware through the UART interface is not the best/preferred solution. If this would be an acceptable long term solution, Keysight would do it by themselves. Instead they are always changing the expensive board, lets assume they know why.
b) My scope did NOT show the usual LED-cycle boot loop too. Nevertheless it was repaired for free.
c) A check through Keysight is free of charge. You may send it in and IF they decide that it is not a NAND problem (=you would have to pay a new board), then you still can decline the repair and you will get your unit back free of charge.

Thus a) + b) + c) shows: they is NO REASON for messing with this by yourself before Keysight had a look at it. Otherwise this might end in an expensive experience.
This is what I did and a I got a scope with a brand new and freshly calibrated main board back. If not, I would have tried to replace the NAND chip.

So my advise: send to Keysight, add a printout of the UBoot prototcol and point to the NAND corruption service note.

[Pinkus]

Mmmmhhh. I was just reading the service note.
At page 1 it says: "NO CHARGE AVAILABLE UNTIL:  31-Oct-2019".
I am wondering if this means that DSOX/MSOX 2K/3K with NAND corruption will not be repaired for free anymore?

[analogRF]

Mmmmhhh. I was just reading the service note.
At page 1 it says: "NO CHARGE AVAILABLE UNTIL:  31-Oct-2019".
I am wondering if this means that DSOX/MSOX 2K/3K with NAND corruption will not be repaired for free anymore?

yes I saw that too 
Still it is a bit hard to believe Keysight will keep replacing the board of these scopes for free until all NAND corrupted boards out there are replaced!
this particular scope is almost 8 years old...but since this line of scopes has not ended yet, maybe they still do that repair for free...I'll call them on Monday.
I will get my LAN board o Tuesday I think and if I dont reach an agreement with Keysight, I'll try that network method

However, that missing 23V rail is still bugging me

[Pinkus]

...I'll call them on Monday.

Been there ....  you will not get a service technician on the line, only administrative people and they will not give you any helpful answers. They will only tell you, you will have to send in the scope.
Good luck!!

[Bud]

You were too quick to load a random replacement image...  The log gives you information on the image length and expected checksum (likely crc32 checksum). Get a few firmware versions  nb0 files and compare their length and checksum against the ones from the log and see if you can find a match. That will be your replacement image to load via Ymodem.

Code: [Select]

...
 Loading image 1 from memory at 0xD0600000
O
BL_IMAGE_TYPE_BIN

X
XXXXOOOOXOOOOOOOOXOXXOOOOOOOOXOOXOXOOOOXXXOOOOOOOOOXOOOOXOXXOXXOXOOOXOXXXXXXOOXXOOOOOOXOOXXOXXOOOERROR: Checksum failure (expected=0x1217115 

computed=0x1217126)
****** Checksum failure on record 36, ABORT!!! ******

Completed file(s):
-------------------------------------------------------------------------------
[0]: Address=0x80361000  Length=0x12F9870  Name="" Target=RAM


[Bud]

Code: [Select]

doesn't support IOCTL_HAL_GET_DEVICE_INFO::SPI_GETBOOTMENAME
Failed to start/configure network.
Time for NANDFLASH to load: 0 ms.
Time for SNANDFLASH to load: 0 ms.
Time for USB Hard Disk Drive to load: 0 ms.
Starting ProcessStartupFolder
ProcessStartupFolder: \usb\Startup\
                                   running \usb\Startup\infiniivision.lnk...

Based on the log, the FPGA was initialized fine, and Windows loaded as well. There was an error configuring the LAN so you may not be able to use a LAN card.

[analogRF]

You were too quick to load a random replacement image...  The log gives you information on the image length and expected checksum (likely crc32 checksum). Get a few firmware versions  nb0 files and compare their length and checksum against the ones from the log and see if you can find a match. That will be your replacement image to load via Ymodem.

Code: [Select]

...
 Loading image 1 from memory at 0xD0600000
O
BL_IMAGE_TYPE_BIN

X
XXXXOOOOXOOOOOOOOXOXXOOOOOOOOXOOXOXOOOOXXXOOOOOOOOOXOOOOXOXXOXXOXOOOXOXXXXXXOOXXOOOOOOXOOXXOXXOOOERROR: Checksum failure (expected=0x1217115 

computed=0x1217126)
****** Checksum failure on record 36, ABORT!!! ******

Completed file(s):
-------------------------------------------------------------------------------
[0]: Address=0x80361000  Length=0x12F9870  Name="" Target=RAM


excellent observation !  thanks

it does not match V2.35
since it is smaller than the size of 2.35, I guess it must be 2.30 perhaps?
I'll try that

EDIT: it matches the size of nk.nb0 file for V2.30 (which also makes sense based on the manufacturing date of this scope if we assume the owner never bothered to upgrade firmware)
I'll try that...but i have my doubts...

[analogRF]

Code: [Select]

doesn't support IOCTL_HAL_GET_DEVICE_INFO::SPI_GETBOOTMENAME
Failed to start/configure network.
Time for NANDFLASH to load: 0 ms.
Time for SNANDFLASH to load: 0 ms.
Time for USB Hard Disk Drive to load: 0 ms.
Starting ProcessStartupFolder
ProcessStartupFolder: \usb\Startup\
                                   running \usb\Startup\infiniivision.lnk...

Based on the log, the FPGA was initialized fine, and Windows loaded as well. There was an error configuring the LAN so you may not be able to use a LAN card.

i am still thinking it has something to do with that missing 23V rail. which is coincidentally an inch above the LAN card connector!
not saying that means anything but...

EDIT: no, the 23V rail comes to life later on