Agilent N9320B stuck at boot: accessed boot prompt and need advice

[analogRF]

recently I got this N9320B in an auction (supposedly came from a university) that does not boot. It gets stuck at the Keysight splash screen (after passing the Agilent splash screen)
There was nothing on the USB and LAN connections at the back either (were not detected).
So I opened it up and first all voltages on the base motherboard are fine (clearly marked test points ). So I got the main cpu board out, which also contains the entire IF and RBW filters and DSP sections of the SA, took it out of its shield (after unscrewing a zillion screws) and installed it back so I can do some limited measurements on the board. Still access is a bit limited due to the PSU being close to it.
Attached pictures show the CPU board that Agilent call it DIF board, by the way.

There are quite a number of DC-DC converters and voltage regulators on this board which I identified and measured all of them. ALL are OK 

The battery ONLY connects to a DS1390 RTC (presumably for keeping the time/date) It does not power any memory as far as I can tell

I found a MAX232 chip and observed some activities on the pin 11 (T1 input) and on the corresponding pin 14 (T1 out). So I connected my Bus Pirate to pins 11 and 12 (on the TTL side) of the MAX232 and set up UART connection with 115200 baud and bingo  I got the boot prompt.
Here is what happens:

Code: [Select]

                            VxWorks System Boot


Copyright 1984-2004  Wind River Systems, Inc.





CPU: Sharp lh7a404
Version: VxWorks5.5.1
BSP version: 1.0/0
Creation date: Oct  1 2008, 10:51:53




Press any key to stop auto-boot...
 0
auto-booting...


boot device          : tffs=0,0
unit number          : 0
processor number     : 0
host name            : host
file name            : /tffs0/VxWorks
inet on ethernet (e) : 192.168.0.111
host inet (h)        : 192.168.0.100
user (u)             : target
ftp password (pw)    : target
flags (f)            : 0x8
other (o)            : csnet

Attaching to TFFS... done.
Loading /tffs0/VxWorks...3414320
Starting at 0xc0001000...

Attaching interface lo0... done
Attached IPv4 interface to csnet unit 0
USB2 Host Stack Initialized.
USB Hub Driver Initialized
USBD  Wind River Systems, Inc. 560 Initialized
1 OHCI Controller found.
Waiting to attach to USBD...Done.

Adding 6135 symbols for standalone.
0xc1effe68 (tRootTask): usbBulkDevInit() returned OK


                VxWorks

Copyright 1984-2002  Wind River Systems, Inc.

            CPU: Sharp lh7a404
   Runtime Name: VxWorks
Runtime Version: 5.5.1
    BSP version: 1.0/0
        Created: Feb  1 2013, 13:54:17
  WDB Comm Type: WDB_COMM_END
            WDB: Ready.

The flash is: S29GL512P
usbTMCDevInit() returned OK.
-> initkey semtake semabcINT failed
taskSpawn keyTask succsess
taskSpawn powerTask succsess
taskSpawn GpibSendData succsess
taskSpawn GpibSendData succsess
uglEpson13506_16BitDevCreate...
uglEpson13506_16BitDevCreate...OK
uglEpsonModeSet...
uglEpsonSetRegs...mode=0
i=0,reg[1,0]
i=1,reg[508,0]
i=2,reg[4,0]
i=3,reg[8,0]
i=4,reg[16,1]
i=5,reg[20,18]
i=6,reg[24,0]
i=7,reg[28,2]
i=8,reg[30,1]
i=9,reg[32,1]
i=10,reg[33,0]
i=11,reg[42,1]
i=12,reg[43,1]
i=13,reg[48,37]
i=14,reg[49,0]
i=15,reg[50,79]
i=16,reg[52,3]
i=17,reg[53,1]
i=18,reg[54,11]
i=19,reg[56,223]
i=20,reg[57,1]
i=21,reg[58,44]
i=22,reg[59,10]
i=23,reg[60,1]
i=24,reg[64,5]
i=25,reg[65,0]
i=26,reg[66,0]
i=27,reg[67,0]
i=28,reg[68,0]
i=29,reg[70,128]
i=30,reg[71,2]
i=31,reg[72,0]
i=32,reg[74,25]
i=33,reg[75,60]
i=34,reg[80,79]
i=35,reg[82,19]
i=36,reg[83,1]
i=37,reg[84,11]
i=38,reg[86,223]
i=39,reg[87,1]
i=40,reg[88,43]
i=41,reg[89,9]
i=42,reg[90,1]
i=43,reg[91,24]
i=44,reg[96,5]
i=45,reg[98,0]
i=46,reg[99,0]
i=47,reg[100,0]
i=48,reg[102,128]
i=49,reg[103,2]
i=50,reg[104,0]
i=51,reg[106,59]
i=52,reg[107,60]
i=53,reg[112,0]
i=54,reg[113,1]
i=55,reg[114,0]
i=56,reg[115,0]
i=57,reg[116,0]
i=58,reg[117,0]
i=59,reg[118,0]
i=60,reg[119,0]
i=61,reg[120,0]
i=62,reg[122,31]
i=63,reg[123,63]
i=64,reg[124,31]
i=65,reg[126,0]
i=66,reg[128,0]
i=67,reg[129,1]
i=68,reg[130,0]
i=69,reg[131,0]
i=70,reg[132,0]
i=71,reg[133,0]
i=72,reg[134,0]
i=73,reg[135,0]
i=74,reg[136,0]
i=75,reg[138,31]
i=76,reg[139,63]
i=77,reg[140,31]
i=78,reg[142,0]
i=79,reg[256,0]
i=80,reg[257,0]
i=81,reg[258,0]
i=82,reg[259,0]
i=83,reg[260,0]
i=84,reg[261,0]
i=85,reg[262,0]
i=86,reg[264,0]
i=87,reg[265,0]
i=88,reg[266,0]
i=89,reg[268,0]
i=90,reg[269,0]
i=91,reg[272,0]
i=92,reg[273,0]
i=93,reg[274,0]
i=94,reg[275,0]
i=95,reg[276,0]
i=96,reg[277,0]
i=97,reg[280,0]
i=98,reg[281,0]
i=99,reg[480,0]
i=100,reg[482,0]
i=101,reg[496,0]
i=102,reg[497,0]
i=103,reg[500,0]
i=104,reg[508,1]
uglEpsonModeSet...OK
0xc1effe68 (tRootTask): memPartFree: invalid block 0x2c6c6c75 in partition 0xc023a3f8.
0xc1effe68 (tRootTask): memPartFree: invalid block 0xc14b8d1c in partition 0xc023a3f8.
0xc1effe68 (tRootTask): memPartFree: invalid block 0x8000011c in partition 0xc023a3f8.
0xc1effe68 (tRootTask): memPartFree: invalid block 0xc14b8934 in partition 0xc023a3f8.

Undefined instruction
Exception address: 0xc1791854
Current Processor Status Register: 0x00000010
Task: 0xc1effe68 "tRootTask"



I cannot quite figure out what my problem could be  I am  an absolute idiot when it comes to these embedded system stuff. Maybe some experts out there can interpret the results and tell me how to proceed...

My guess is that either the memory (SRAM) is bad which would be a very good news or the FLASH ROM is corrupted which for me is perhaps the end of this adventure 

Some further information: the main processor has 80KB of on chip SRAM and a boot ROM in it too.

after the boot stopped I pressed enter and got this -> prompt and ran some information gathering commands which you can see here:

Code: [Select]

.....
Undefined instruction
Exception address: 0xc1791854
Current Processor Status Register: 0x00000010
Task: 0xc1effe68 "tRootTask"

-> moduleShow

MODULE NAME     MODULE ID  GROUP #    TEXT START DATA START  BSS START
--------------- ---------- ---------- ---------- ---------- ----------
     usb488.out 0xc18d74e8          1 0xc179b350 0xc18b4814 0xc18c5480
         SA.out 0xc14b8f6c          2          0          0          0
value = 0 = 0x0
-> ls
0xc18f1940 (tShell): memPartAlloc: block too big - 99 in partition 0xc023a3f8.
value = -1 = 0xffffffff


-> version
VxWorks (for Sharp lh7a404) version 5.5.1.
Kernel: WIND version 2.6.
Made on Feb  1 2013, 13:54:17.
Boot line:
tffs=0,0(0,0)host:/tffs0/VxWorks h=192.168.0.100 e=192.168.0.111 f=0x8 u=target pw=target o=csnet
value = 109 = 0x6d = 'm'
->
->
-> devs
drv name
  0 /null
  1 /tyCo/0
  1 /tyCo/1
  3 /tffs0
  5 host:
  6 /bpf/dhcpc/0
  6 /bpf/dhcpc/1
  6 /bpf/dhcpc/2
  6 /bpf/dhcpc/3
  6 /bpf/dhcpc-arp
  9 /vio
value = 25 = 0x19
->
->
-> lkup
Number of Symbols   : 8464
Symbol Mutex Id     : 0xc18f4bb4
Symbol Hash Id      : 0xc18f438c
Symbol memPartId    : 0xc023a3f8
Name Clash Policy   : Allowed
value = 0 = 0x0
->
->
-> taskShow

  NAME        ENTRY       TID    PRI   STATUS      PC       SP     ERRNO  DELAY
---------- ------------ -------- --- ---------- -------- -------- ------- -----
tRootTask  usrRoot      c1effe68   0 SUSPEND    c1791854        0  110003     0
value = 0 = 0x0
->

It seems the firmware is made in 2013 but agilent has had firmware updates in 2014, 2015, and the last one was 2017. So my guess is that it was not corrupted during a firmware update (I know this device was decommissioned not too long ago definitely not 5-6 years ago)

also when I press a key and stop the auto boot, I get these after typing ? help command:

Code: [Select]

Version: VxWorks5.5.1
BSP version: 1.0/0
Creation date: Oct  1 2008, 10:51:53




Press any key to stop auto-boot...
 1
[VxWorks Boot]: ?

 ?                     - print this list
 @                     - boot (load and go)
 p                     - print boot params
 c                     - change boot params
 l                     - load boot file
 g adrs                - go to adrs
 d adrs[,n]            - display memory
 m adrs                - modify memory
 f adrs, nbytes, value - fill memory
 t adrs, adrs, nbytes  - copy memory
 e                     - print fatal exception
 v                     - print boot logo with version
 n netif               - print network interface device address
 $dev(0,procnum)host:/file h=# e=# b=# g=# u=usr [pw=passwd] f=#
                           tn=targetname s=script o=other
 boot device: tffs=drive,removable     file name: /tffs0/vxWorks
 Boot flags:
   0x02  - load local system symbols
   0x04  - don't autoboot
   0x08  - quick autoboot (no countdown)
   0x20  - disable login security
   0x40  - use bootp to get boot parameters
   0x80  - use tftp to get boot image
   0x100 - use proxy arp

available boot devices:Enhanced Network Devices
 csnet0 tffs
[VxWorks Boot]:

any advice is highly appreciated

EDIT: funny, someone spelled success wrong

[analogRF]

any hint or suggestion?

I am planning to replace the SRAM chips first just out of desperation and see what happens. ordered the chip from China...

but if it is the flash EPROM being corrupted, is there any way to fix it?
is there any way to program the firmware into it? I think the boot loader is inside the CPU....

any help would be wonderful...

[analogRF]

almost 100 reads and no suggestion 

it seems I have the option of changing boot parameters and perhaps force it to boot from tftp

does anybody know if and how this is possible? do I need a boot image (like a .bin file) of the whole flash eprom on a local tftp server?
can I do anything with the firmeware downloaded from keysight?

[TheSteve]

Having no experience with a N9320B or VxWorks to any real extent it is hard to offer advice. It would be nice to know what the boot messages look like on a functioning unit. It does seem quite possible there is a ram issue or it is flash corruption.
If it was mine I would certainly be experimenting with the tftp option to recovery it.

[analogRF]

Having no experience with a N9320B or VxWorks to any real extent it is hard to offer advice. It would be nice to know what the boot messages look like on a functioning unit. It does seem quite possible there is a ram issue or it is flash corruption.
If it was mine I would certainly be experimenting with the tftp option to recovery it.

yes that's what I am thinking about but how? I have seen people reviving modem's with tftp booting but they have a single .bin file for the firmware
for now i am waiting for the SRAM to arrive and try to replace them...but flash corruption seems more likely 

[YetAnotherTechie]

Don't give up yet, that is nor flash and your system boots, so you can get a new chip and program it with the current contents, and then do a firmware upgrade.
What firmware versions from keysight do you have? Do you have the same that is installed? i only found the latest 2017-04-13    B.03.61
I would do a copy of the entire memory using the commands you found, and compare it to a keysight upgrade file.
I would remove the chip and back it up anyway.
If i were confident that the copy i made is the same as in the physical chip, i would try a firmware update (DON'T try this before making a backup of the contents of the actual chip, even if you have to remove it)
Since it boots and the usb subsystem is listed as working in your logs, it might recognize a usb flash drive prepared for the firmware update.
Try keeping a key pressed while booting, for example preset.
Ask keysight if there is a full reset key while booting, or a bad firmware recovery procedure?

[analogRF]

Don't give up yet, that is nor flash and your system boots, so you can get a new chip and program it with the current contents, and then do a firmware upgrade.
What firmware versions from keysight do you have? Do you have the same that is installed? i only found the latest 2017-04-13    B.03.61
I would do a copy of the entire memory using the commands you found, and compare it to a keysight upgrade file.
I would remove the chip and back it up anyway.
If i were confident that the copy i made is the same as in the physical chip, i would try a firmware update (DON'T try this before making a backup of the contents of the actual chip, even if you have to remove it)
Since it boots and the usb subsystem is listed as working in your logs, it might recognize a usb flash drive prepared for the firmware update.
Try keeping a key pressed while booting, for example preset.
Ask keysight if there is a full reset key while booting, or a bad firmware recovery procedure?

i am not giving up at all. i didn't get what you meant here

...so you can get a new chip and program it with the current contents, and then do a firmware upgrade

if the current content is corrupted somehow, then the new chip will have the same issue, no? maybe I misunderstood...

I only have the latest firmware from keysight website. If you unzip it it has a bunch of .out modules and .txt files which are config files and a single VxWorks  file with no extension. plus a couple of directories including all the texts that the SA shows on screen and some .jpg images

trying to reflash it from usb is a good point. I didn't try because it is not the normal way of firmware update (it is done through the system menus when it is working) but I also suspect that there might be a back door that let me reflash it at power on. The problem is that even though the boot sequence showed LAN and USB were enabled, when I connected them to my computer (back USB host connector) or to my LAN I got no activity
by that I mean Agilent IO and connection expert could not find the instrument. So maybe my way of testing was wrong. But I got no new IP listed on my network so I could not even ping it.

I have asked help from Keysight support but no answer yet. Although i didn't ask specifically about a trick to firmware update at boot. That would be wonderful...I will ask them when they answer my request and conversation gets going

[YetAnotherTechie]

I tought you would give up because of this:

...but flash corruption seems more likely 

Your flash is a NOR chip, it has a one-to-one relation between contents inside and what it outputs, NAND flash has contents + error correction and even if you had the firmware it would be much harder to figure out how to program a new chip.
The analyser has an embedded file system, is not a linear program like small micros, it's like your hard disk, and your logs say that there is an error in one of the partitions (a logical division of that disk)
Since your system boots, a firmware upgrade might do something similar to formatting the disk and writing it entirely again, which might work, if there is no longer a hardware fault in the chip.
It might fail even if there is no fault, only corruption. Some embedded systems rewrite the partition scheme, some don't. In that case you need to make an image of the file system, and load it on linux, analyse the partition scheme, etc, etc, a lot of work, (think data-recovery) and i'm a beginner at that.
What i suggested was to create a dump of the chip you have and compare it to some of the files in the keysight upgrade, more specifically the VxWorks file, of course they won't match unless you have the same version, but should be similar.
Your SA.out file seems corrupted, but is it the file, or the partition? who knows...
Try pinging 192.168.0.100 or 192.168.0.111, for this to work you need to set up your pc to have a fixed IP in the range 192.168.0.1xx and connect directly to the SA.
if it doesn't ping you should still try a tftp server program to connect. Don't write anything until you hava a backup of your flash.
Until you have a copy of the flash, there's not much anyone can do to help you. And if you loose the working copy you have then no one can help you, because you probably loose instrument specific data. Good luck, and patience!

[analogRF]

the only programmer that I have is TL866 II which seems unable to read or write TSOP56 flash chips :-(

any suggestion for a programmer that is not too expensive?

[analogRF]

isnt there any online command that can be used to dump the whole firmware from the system?

[YetAnotherTechie]

I also only have the tl866. My solution would be to use an adapter board and read the chip 4 or 8 times, as needed, by setting the unconnected high address lines manually.
Try the d adrs[,n] command
d 0x00000000, 0x3FFFFFF post results
d 0xc0000000, 0x3FFFFFF post results
with SA off, plug an USB drive, empty except for one file named test.txt, drive should be one of keysight recommended list, or at most 1gb,
boot and post the result of the same steps you showed on your first post.
Are there any other flash chips inside?

[YetAnotherTechie]

try:
memShow - Display allocated/free memory
memPartS0how - Show memory partition
semShow - Show info about semaphor
i - Display synopsis of all tasks
taskshow - List info about the tasks from TCB
pwd - Present working directory
ls - list the directory

[analogRF]

I was looking inside that VxWorks file that comes with the downloaded firmware, and I noticed that these texts are in there:
"WIND version 2.6", "VxWorks version 5.5.1", "Feb  1 2013, 13:54:17"

These are exactly what I get when I run the "version" command when I am connected to the instrument (see my first post) so at least the OS part of the firmware seems to be the same as the latest version on keysight website but this might have also been used in older versions of the firmware, too. I don't know the version of SA software modules...
another thing I noticed is that N9320B is mentioned inside the vxworks file! does it mean the OS is customized for this instrument? I thought the vxworks is a general OS.

i don;t know why I cannot run the "ls" command to see the content of current directory. See the error that I got in my first post...

you can see that there are two boot devices, one is the tffs which I assume is the file system residing on the flash and the other one is csnet0

I wonder how I can change my boot parameters to use csnet0 for booting and how I can use that to boot from a location on network? is that used for booting through tftp? I would like to try this route and maybe bypass the flash chip and boot the system. but i dont know how to set it all up
I am thinking just putting the raw unpacked firmware somewhere and point the instrument to boot from that directory on, for example, on my laptop. Can somebody help me to set it up?

[analogRF]

try:
memShow - Display allocated/free memory
memPartS0how - Show memory partition
semShow - Show info about semaphor
i - Display synopsis of all tasks
taskshow - List info about the tasks from TCB
pwd - Present working directory
ls - list the directory

please see my first post. I did all these commands and reported the results. some of them did not work and were "unknown" so I didnt report them
but i did try all these commands that you just mentioned except the "i" command

[YetAnotherTechie]

I'm new to vxworks, but from what i read, if it could recognize pendrive, then loading a file would be much easier. It would show up on the ""available boot devices:" line.
Try "ls" on other devices/directories to see if it's the command or the folders fault.

[analogRF]

I also notice that in my boot log I have this line : "Loading /tffs0/VxWorks...3414320"
but the size of the vxworks file that is in the firmware package is 3142892
since I am now pretty sure the same vxworks file is used in the current system (see post #12) then, does it mean it is de-compressed when loaded
or does it mean it is corrupted and the size is wrong?!

[YetAnotherTechie]

I also notice that in my boot log I have this line : "Loading /tffs0/VxWorks...3414320"
but the size of the vxworks file that is in the firmware package is 3142892
since I am now pretty sure the same vxworks file is used in the current system (see post #12) then, does it mean it is de-compressed when loaded
or does it mean it is corrupted and the size is wrong?!

From what i read if it was compressed it would have a name with an extension, so i've no idea if it's good or bad.
Try
tffsShow
tffsShowAll
usrFdiskPartShow "/tffs0"
dosFsShow "/tffs0"
alternatives /tffs0/ /tffs /tffs/
L
ll

[analogRF]

some update and more disappointment 
I put a FAT32 formatted USB drive into the front USB port and as you can see here, the instrument recognized it and i believe it is mounted as /db0 device.

Code: [Select]

                            VxWorks System Boot


Copyright 1984-2004  Wind River Systems, Inc.





CPU: Sharp lh7a404
Version: VxWorks5.5.1
BSP version: 1.0/0
Creation date: Oct  1 2008, 10:51:53




Press any key to stop auto-boot...
 0
auto-booting...


boot device          : tffs=0,0
unit number          : 0
processor number     : 0
host name            : host
file name            : /tffs0/VxWorks
inet on ethernet (e) : 192.168.0.111
host inet (h)        : 192.168.0.100
user (u)             : target
ftp password (pw)    : target
flags (f)            : 0x8
other (o)            : csnet

Attaching to TFFS... done.
Loading /tffs0/VxWorks...3414320
Starting at 0xc0001000...

Attaching interface lo0... done
Attached IPv4 interface to csnet unit 0
USB2 Host Stack Initialized.
USB Hub Driver Initialized
USBD  Wind River Systems, Inc. 560 Initialized
1 OHCI Controller found.
Waiting to attach to USBD...Done.

Adding 6135 symbols for standalone.
0xc1effe68 (tRootTask): usbBulkDevInit() returned OK


                VxWorks

Copyright 1984-2002  Wind River Systems, Inc.

            CPU: Sharp lh7a404
   Runtime Name: VxWorks
Runtime Version: 5.5.1
    BSP version: 1.0/0
        Created: Feb  1 2013, 13:54:17
  WDB Comm Type: WDB_COMM_END
            WDB: Ready.

The flash is: S29GL512P
usbTMCDevInit() returned OK.
-> Found Bulk Device with 1 Logical Units at node 2
initkey semtake semabcINT failed
taskSpawn keyTask succsess
taskSpawn powerTask succsess
taskSpawn GpibSendData succsess
taskSpawn GpibSendData succsess
uglEpson13506_16BitDevCreate...
uglEpson13506_16BitDevCreate...OK
uglEpsonModeSet...
uglEpsonSetRegs...mode=0
i=0,reg[1,0]
i=1,reg[508,0]
i=2,reg[4,0]
i=3,reg[8,0]
i=4,reg[16,1]
i=5,reg[20,18]
i=6,reg[24,0]
i=7,reg[28,2]
i=8,reg[30,1]
i=9,reg[32,1]
i=10,reg[33,0]
i=11,reg[42,1]
i=12,reg[43,1]
i=13,reg[48,37]
i=14,reg[49,0]
i=15,reg[50,79]
i=16,reg[52,3]
i=17,reg[53,1]
i=18,reg[54,11]
i=19,reg[56,223]
i=20,reg[57,1]
i=21,reg[58,44]
i=22,reg[59,10]
i=23,reg[60,1]
i=24,reg[64,5]
i=25,reg[65,0]
i=26,reg[66,0]
i=27,reg[67,0]
i=28,reg[68,0]
i=29 ,reg[70,128]
i=30,reg[71,2]
i=31,reg[72,0] /bd0 Bulk device is created
i=32,reg[74,25]
i=33,reg[75,60]
i=34,reg[80,79]
i=35,reg[82,19]
i=36,reg[83,1]
i=37,reg[84,11]
i=38,reg[86,223]
i=39,reg[87,1]
i=40,reg[88,43]
i=41,reg[89,9]
i=42,reg[90,1]
i=43,reg[91,24]
i=44,reg[96,5]
i=45,reg[98,0]
i=46,reg[99,0]
i=47,reg[100,0]
i=48,reg[102,128]
i=49,reg[103,2]
i=50,reg[104,0]
i=51,reg[106,59]
i=52,reg[107,60]
i=53,reg[112,0]
i=54,reg[113,1]
i=55,reg[114,0]
i=56,reg[115,0]
i=57,reg[116,0]
i=58,reg[117,0]
i=59,reg[118,0]
i=60,reg[119,0]
i=61,reg[120,0]
i=62,reg[122,31]
i=63,reg[123,63]
i=64,reg[124,31]
i=65,reg[126,0]
i=66,reg[128,0]
i=67,reg[129,1]
i=68,reg[130,0]
i=69,reg[131,0]
i=70,reg[132,0]
i=71,reg[133,0]
i=72,reg[134,0]
i=73,reg[135,0]
i=74,reg[136,0]
i=75,reg[138,31]
i=76,reg[139,63]
i=77,reg[140,31]
i=78,reg[142,0]
i=79,reg[256,0]
i=80,reg[257,0]
i=81,reg[258,0]
i=82,reg[259,0]
i=83,reg[260,0]
i=84,reg[261,0]
i=85,reg[262,0]
i=86,reg[264,0]
i=87,reg[265,0]
i=88,reg[266,0]
i=89,reg[268,0]
i=90,reg[269,0]
i=91,reg[272,0]
i=92,reg[273,0]
i=93,reg[274,0]
i=94,reg[275,0]
i=95,reg[276,0]
i=96,reg[277,0]
i=97,reg[280,0]
i=98,reg[281,0]
i=99,reg[480,0]
i=100,reg[482,0]
i=101,reg[496,0]
i=102,reg[497,0]
i=103,reg[500,0]
i=104,reg[508,1]
uglEpsonModeSet...OK
0xc1effe68 (tRootTask): memPartFree: invalid block 0xc1487240 in partition 0xc023a3f8.
0xc1effe68 (tRootTask): memPartFree: invalid block 0x8000011c in partition 0xc023a3f8.
0xc1effe68 (tRootTask): memPartFree: invalid block 0xc1486de8 in partition 0xc023a3f8.

Undefined instruction
Exception address: 0xc1486e04
Current Processor Status Register: 0x00000010
Task: 0xc1effe68 "tRootTask"

after it stops, I executed these commands at the prompt and here are the results:

Code: [Select]

-> devs
drv name
  0 /null
  1 /tyCo/0
  1 /tyCo/1
  3 /tffs0
  5 host:
  6 /bpf/dhcpc/0
  6 /bpf/dhcpc/1
  6 /bpf/dhcpc/2
  6 /bpf/dhcpc/3
  6 /bpf/dhcpc-arp
  9 /vio
  3 /bd0
value = 25 = 0x19
-> moduleShow

MODULE NAME     MODULE ID  GROUP #    TEXT START DATA START  BSS START
--------------- ---------- ---------- ---------- ---------- ----------
     usb488.out 0xc18d7398          1 0xc179b200 0xc18b46c4 0xc18c5330
         SA.out 0xc1487490          2          0          0          0
value = 0 = 0x0
-> i

  NAME        ENTRY       TID    PRI   STATUS      PC       SP     ERRNO  DELAY
---------- ------------ -------- --- ---------- -------- -------- ------- -----
tRootTask  usrRoot      c1effe68   0 SUSPEND    c1486dec        0  110003     0
tExcTask   excTask      c1bf25b4   0 PEND       c01ce7fc c1bf24bc       0     0
tLogTask   logTask      c1befc38   0 PEND       c01ce7fc c1befb50       0     0
tNbioLog   c012443c     c1bed5a8   0 PEND       c0063ea0 c1bed4f8       0     0
tShell     shell        c18f1940   1 READY      c0188b68 c18f15d4  110001     0
tWdbTask   wdbTask      c18f3b50   3 PEND       c0063ea0 c18f3a84       0     0
tUSBVbusMonc1892474     c179b058   5 DELAY      c0187a90 c179af8c       0     2
tUSBEP0Intrc1892474     c14ddcb0   5 PEND       c0063ea0 c14ddbfc       0     0
tUSBTargIntc1892474     c14e5e58  10 PEND       c0063ea0 c14e5da4       0     0
tNetTask   netTask      c1bc56b8  50 PEND       c0063ea0 c1bc5654       0     0
tPortmapd  portmapd     c1b17664  54 PEND       c0063ea0 c1b17490  3d0002     0
tTelnetd   telnetd      c1b14ba0  55 PEND       c0063ea0 c1b14abc       0     0
tFtpdTask  c0080a00     c1b126bc  55 PEND       c0063ea0 c1b1257c       0     0
tDhcpcStatec0079274     c1b1a174  56 PEND       c0063ea0 c1b1a0ec       0     0
tDhcpcReadTdhcpcRead    c1b18bec  56 PEND       c0063ea0 c1b18a08  3d0002     0
tUglInput  uglInputTask c148c980  60 PEND+T     c0063ea0 c148c65c  3d0002    60
GPIBINT    GpibIntHande c14c6a88  90 PEND       c0063ea0 c14c6a24       0     0
PowerMonitopowerTask    c14d5b08  99 PEND       c0063ea0 c14d5aac       0     0
tTffsPTask flPollTask   c1bebce8 100 DELAY      c0187a90 c1bebc70       0     5
t1         c013fa34     c190f8b4 100 PEND       c01ce7fc c190f788       0     0
usbOhciIsr c0167dd8     c1906e20 100 PEND       c0063ea0 c1906dbc       0     0
BusM A     c014abc0     c1904bf0 100 DELAY      c0187a90 c1904b6c       0     1
BULK_CLASS c013fa34     c19024a4 100 PEND       c01ce7fc c1902278       0     0
BULK_CLASS_c013fa34     c18fdd88 100 PEND       c01ce7fc c18fdcc8       0     0
tBulkClnt  c0018e38     c18f9af8 100 PEND       c0063ea0 c18f9a70       0     0
usbTMCLib  c013fa34     c18e1134 100 PEND       c01ce7fc c18e0f08       0     0
usbTMCLib_Ic013fa34     c18dca18 100 PEND       c01ce7fc c18dc958       0     0
keytask    keypadRead   c18ce964 100 PEND       c0063ea0 c18ce8f4       0     0
GPIBOUTPUT GpibSendData c14ce160 106 PEND       c0063ea0 c14ce0fc       0     0
tDcacheUpd dcacheUpd    c1bc914c 250 DELAY      c0187a90 c1bc90b4       0     9
value = 0 = 0x0
->
->
->
-> pwd
host:
value = 6 = 0x6
->
->
-> ls
0xc18f1940 (tShell): memPartAlloc: block too big - 99 in partition 0xc023a3f8.
value = -1 = 0xffffffff
->
-> cd "/db0"
0xc18f1940 (tShell): memPartAlloc: block too big - 5 in partition 0xc023a3f8.
not enough memory for new string.
->
->
-> cd "/tffs0"
0xc18f1940 (tShell): memPartAlloc: block too big - 7 in partition 0xc023a3f8.
not enough memory for new string.
->
->
-> cd /db0
syntax error
->
-> ll
0xc18f1940 (tShell): memPartAlloc: block too big - 99 in partition 0xc023a3f8.
value = -1 = 0xffffffff
->
-> fdisk
undefined symbol: fdisk
-> taskShow

  NAME        ENTRY       TID    PRI   STATUS      PC       SP     ERRNO  DELAY
---------- ------------ -------- --- ---------- -------- -------- ------- -----
tShell     shell        c18f1940   1 READY      c0188b68 c18f15d4  110001     0
value = 0 = 0x0
->
->
-> ti tRootTask

  NAME        ENTRY       TID    PRI   STATUS      PC       SP     ERRNO  DELAY
---------- ------------ -------- --- ---------- -------- -------- ------- -----
tRootTask  usrRoot      c1effe68   0 SUSPEND    c1486dec        0  110003     0

stack: base 0xc1effe68  end 0xc1efc018  size 15940  high 14400  margin 1540

options: 0x6
VX_UNBREAKABLE      VX_DEALLOC_STACK

VxWorks Events
--------------
Events Pended on    : Not Pended
Received Events     : 0x0
Options             : N/A

r0     =        4   r1     = c04feea0   r2     = 807c3d3a   r3     = c1efb008
r4     =        0   r5     = c1effb6c   r6     = c1487490   r7     =        0
r8     = c023a3f8   r9     =        0   r10    = 74732df8   r11/fp = 74732df8
r12/ip =        0   r13/sp =        0   r14/lr = ffffffff   pc     = c1486dec
cpsr   =       10

Undefined instructionvalue =
0Exception address: 0x = 0xc1486dec
Current Processor Status Register: 0x000000010

->checkStack
  NAME        ENTRY        TID     SIZE   CUR  HIGH  MARGIN
------------ ------------ -------- ----- ----- ----- ------
tRootTask    usrRoot      c1effe68 15940  1168 14400   1540
tExcTask     excTask      c1bf25b4  7988   248  1008   6980
tLogTask     logTask      c1befc38  4988   232   908   4080
tNbioLog     0x00c012443c c1bed5a8  4988   176   236   4752
tShell       shell        c18f1940 64860   876  3924  60936
tWdbTask     wdbTask      c18f3b50  8180   204   264   7916
tUSBVbusMon  0x00c1892474 c179b058 32756   204   260  32496
tUSBEP0Intr  0x00c1892474 c14ddcb0 32756   180   240  32516
tUSBTargIntr 0x00c1892474 c14e5e58 32752   180   240  32512
tNetTask     netTask      c1bc56b8  9988   100   416   9572
tPortmapd    portmapd     c1b17664  9988   468   652   9336
tTelnetd     telnetd      c1b14ba0  7988   228   288   7700
tFtpdTask    0x00c0080a00 c1b126bc  6132   320   380   5752
tDhcpcStateT 0x00c0079274 c1b1a174  4984   136   196   4788
tDhcpcReadTa dhcpcRead    c1b18bec  4984   484   668   4316
tUglInput    uglInputTask c148c980  4084   804   864   3220
GPIBINT      GpibIntHande c14c6a88 29992   100   160  29832
PowerMonitor powerTask    c14d5b08 30704    92   152  30552
tTffsPTask   flPollTask   c1bebce8  2036   120   296   1740
t1           0x00c013fa34 c190f8b4 16380   300   732  15648
usbOhciIsr   0x00c0167dd8 c1906e20  8180   100   400   7780
BusM A       0x00c014abc0 c1904bf0  8184   132   728   7456
BULK_CLASS   0x00c013fa34 c19024a4 16372   556  1772  14600
BULK_CLASS_I 0x00c013fa34 c18fdd88 16368   192   252  16116
tBulkClnt    0x00c0018e38 c18f9af8 19988   136   948  19040
usbTMCLib    0x00c013fa34 c18e1134 16372   556   616  15756
usbTMCLib_IR 0x00c013fa34 c18dca18 16368   192 14332   2036
keytask      keypadRead   c18ce964 30712   112   200  30512
GPIBOUTPUT   GpibSendData c14ce160 29988   100   200  29788
tDcacheUpd   dcacheUpd    c1bc914c  4988   152   328   4660
INTERRUPT                           1792     0   304   1488 - SVC -
                                     256     0    24    232 - IRQ -
value = 43 = 0x2b = '+'
-> memShow
undefined symbol: memShow
-> memPartShow
undefined symbol: memPartShow
->
-> chkdsk "/tffs0"
0xc18f1940 (tShell): memPartAlloc: block too big - 7 in partition 0xc023a3f8.
not enough memory for new string.

-> ioHelp

cd        "path"             Set current working path
pwd                            Print working path
ls        ["wpat"[,long]]    List contents of directory
ll        ["wpat"]           List contents of directory - long format
lsr       ["wpat"[,long]]    Recursive list of directory contents
llr       ["wpat"]           Recursive detailed list of directory
rename    "old","new"      Change name of file
copy      ["in"][,"out"]   Copy in file to out file (0 = std in/out)
cp        "wpat","dst"      Copy many files to another dir
xcopy     "wpat","dst"      Recursively copy files
mv        "wpat","dst"      Move files into another directory
xdelete   "wpat"             Delete a file, wildcard list or tree
attrib    "path","attr"    Modify file attributes
xattrib   "wpat","attr"    Recursively modify file attributes
chkdsk    "device", L, V     Consistency check of file system
diskInit  "device"           Initialize file system on disk
diskFormat "device"          Low level and file system disk format

"attr" contains one or more of: " + - A H S R" characters

"wpat" may be name of a file, directory or wildcard pattern
 in which case "dst" must be a directory name
chkdsk() params: L=0, check only, L=2, check and fix, V=0x200 verbose

value = 1 = 0x1

-> chkdsk "/tffs0"
0xc18f1940 (tShell): memPartAlloc: block too big - 7 in partition 0xc023a3f8.
not enough memory for new string.
-> chkdsk "/db0"
0xc18f1940 (tShell): memPartAlloc: block too big - 5 in partition 0xc023a3f8.
not enough memory for new string.

if you compare with what I had before in the first post, you see the difference which is not much except the USB stick is now a new drive
the problem is there is no way I can access or point to not only the USB device (/db0) but to any part of the file system, when I try commands
like "ll", "ls", "cd" or "chkdsk" etc...for /db0 or /tffs0 I get a memory allocation error. (see above)

but about the disappointment, unfortunately I am no longer able to halt the boot loader and stop the boot process in order to change boot parameters anymore. In the beginning there was a message (after VxWorks title) that "press any key to stop auto-boot..." but that is now disappeared and no matter how fast I pound on the keyboard I am not able to stop auto-booting 
I played with boot parameters and tried (with my limited knowledge) to force it to load the boot file from USB but I was very careful not to play with anything else other than the boot device and boot file name (I did not change the boot flag and it is the same as before) and I reversed everything back but somehow I found out that I am not getting that "press any key to stop auto boot" message and I am not able to stop auto boot...
so setting up tftp is now out of the window...although i dont think it would have helped anyway...

you see, the system loads the VxWorks into memory and runs quite a bit of stuff but my guess is that when it tries to free memory or partition memory in a certain section of memory it gets stuck because it receives garbage perhaps...I think one of the RAM chips might be bad and when it tries to use its address space it gets stuck...i don't know...

[analogRF]

anybody know how I can get that stop auto-booting option back?
can anybody show me a forum that vxworks people hang around and might know this?

I tried google for hours but didnt find anything 
still I dont know how that could have happened. The boot loader as I understand resides inside the cpu
and then it loads the main vxworks file for this system from the flash chip...

[analogRF]

I'm new to vxworks, but from what i read, if it could recognize pendrive, then loading a file would be much easier. It would show up on the ""available boot devices:" line.
Try "ls" on other devices/directories to see if it's the command or the folders fault.

I noticed that it does not show up in the boot devices. In fact I found out that at that time during initial boot up (I assume from the boot loader inside the CPU perhaps) actually the USB device is not known at all (only ethernet is known as csnet0 device)
Later on after loading the VxWorks into memory from the flash chip  and running it, the system initializes USB and finds the drive...

EDIT: I was thinking the same and that got me in trouble because i tried to play with those damn parameters and even though I cannot think of doing anything that might have resulted "fast boot only" but apparently it is what has occurred...boot parameters on scree still show up exactly as before though...

[YetAnotherTechie]

I noticed that the bad memory locations decreased from 4 to 3 right?
So, crazy idea, plug as many devices as you can in usb ports, use a hub if you have to, if it's ram, maybe it will shift the bad memory to a different task and you can do something.

[analogRF]

yes for some reason there are three lines giving allocation error instead of 4 but that doesnt mean anything. These are the same error I get when I try to execute any file/directory command like "cd", "ls" ...
now that I am no longer able to stop the autobooting, reading the boot file from tftp is not possible but, like I said, I dont really think the firmware is corrupted. It is indeed loading the kernel into memory,  the problem however is that it gets corrupted IN the memory I think.
for now I am waiting for the memory chips to show up and try to replace them and see what happens...I am pretty much out of options after that...

I ran the serial number on keysight website and i found out that the instrument had a 3 month replacement guarantee between Oct 2, 2017 to Jan 2, 2018! I know the instrument is older than that (judging by the date codes on the chips, it's probably made around 2011) so perhaps it was refurbished or repaired by Keysight at that time, if that is the case I think they must have upgraded the firmware at that time, too and the last firmware came out 5-6 month before Oct 2017. So my theory is that this thing died one day in the lab (university as the seller told me) not because students tried to upgrade the firmware which they will never do but because one of the SRAM chips gave out...

[analogRF]

for the first time even Keysight tech support didn't bother to get back to me on this at all after 7-8 days now 

[Renate]

Mmm, let me tell you about a recent unrelated experience of mine.

Same sort of setup, different device, different processor.
There's an on-chip ROM bootloader and in flash a secondary bootloader and a system.
The ROM bootloader can boot off the flash or using the iMX SDP protocol over USB, but only if the flash fails.
I accidentally borked the secondary bootloader, but not enough for the ROM bootloader to know that it was bad.
It would start up, load the broken secondary bootloader and fail.
Since it already passed the decision point in the ROM bootloader, I didn't have any access to the SDP over USB.

So, I grounded one of the data/control pins (dunno what it was) on the flash and reset the processor.
The ROM bootloader came up, noticed the inaccessible flash and started running the SDP over USB.
(At that point I removed the ground.)
I loaded in the secondary bootloader over USB, booted the system from flash and used that to rewrite the secondary bootloader in flash.
Everything was back to working.

I couldn't make out exactly what processor that is. I don't know what boot options/protocols the ROM bootloader uses.

[analogRF]

Thanks, Renate for suggestion
based on what I have experienced with this thing, I am pretty sure the boot loader (which I believe is sitting on CPU chip) can only boot from csnet device (internet) or tffs which i think refers to the flash chip. I am pretty sure during the original boot no usb device was initialized and recognized (my usb stick had a led on it,I could see when there was activity) it was only recognized after vxworks was loaded into memory (from flash)

however, now I have lost the ability to stop autobooting which means I must have done something that is written in the config of the on-chip boot loader! seems a bit odd to me that such a thing would be possible or allowed...besides the boot flag still shows 0x8 which is the original one and has not changed. so I cannot figure out why I have lost that option

anyways,  I was hoping I can get it to stop autoboot so I can change the boot settings and perhaps setup a tftp and boot it from tftp. 
now your suggestion might do the trick, if it cannot boot from tffs, perhaps it will stop and give me the boot prompt and then I can change the boot settings.

one question also remains and that is even if I set it up to boot from tftp, the only thing I have available is the firmware files downloaded from keysight. is that enough for the system to boot?

for now I am planning first to replace the SRAM because as I explained, I am still hopeful that this instrument didnt die due to bad firmware update.
if that doesn't work, I will do your trick and try tftp boot