Tektronix DPO 4104 kernel panic

[adrianh]

Hi all. First time post here. Please let me know if I should post to another forum/area.

I recently acquired a not-working DPO 4104. I know nothing of its history. It is in a splash-screen boot-loop. I managed to figure out where the serial port is, and captured the following log:

... [normal stuff]
## Booting image at f0000000 ...
   Image Name:   Linux-2.4.20_mvl31-440ep_eval
   Image Type:   PowerPC Linux Multi-File Image (gzip compressed)
   Data Size:    1441010 Bytes =  1.4 MB
   Load Address: 00000000
   Entry Point:  00000000
   Contents:
   Image 0:  1033953 Bytes = 1009.7 kB
   Image 1:   407042 Bytes = 397.5 kB
   Verifying Checksum ... OK
   Uncompressing Multi-File Image ... OK
cmdline is console=ttyS0,9600 quiet bigphysarea=519 panic=2 root=/dev/mtdblock7 rw mem=131072k
   Loading Ramdisk to 07f2b000, end 07f8e602 ... OK
kernel BUG at page_alloc.c:104!
Oops: Exception in kernel mode, sig: 4
NIP: C0036248 XER: 00000000 LR: C0036248 SP: C01E8820 REGS: c01e8770 TRAP: 0700    Not tainted
MSR: 00009030 EE: 1 PR: 0 FP: 0 ME: 1 IR/DR: 11
DEAR: 00004000, ESR: 08000000
TASK = c01e6960[0] 'swapper' Last syscall: 0
[... goes into reboot loop]

The error condition is flagged in the kernel when pages are being freed, so this looks like completely bogus page, or a double-free.

        if (!VALID_PAGE(page))
                BUG();

This is repeats exactly the same way everytime.

My initial suspicion is that the scope was improperly updated, or the update failed for some reason and the scope is effectively bricked.

There could be a hardware failure I guess (bad DRAM cell(s) since this is happening with a RAM disk?).

Is there anything that can be done at this point? - either more debug or some sort of flash of a new image? A traditional "firmware update" is not possible since the machine cannot boot to any level of functionality.

Thanks for any ideas!

[pcwrangler]

By the boot log it doesn't look like it gets to the firmware update step but you might want to try the "FORCE" firmware method in the manual just to be thorough. Ramdisk load is the last line which is curious. You could be right about a bad flash but I'm afraid it could also be bad RAM that it tries to access after the initial copy.

I wish I had a copy of the flash chip for you to try programming but I haven't been able to find one for my MSO4104. On mine it is a 4M flash chip at U822 but contains more than just the bootloader or it would be a simple task. I'm interested to see if this thread gains any traction because I am in a similar situation. Please update with anything you learn as it would help me and others in the future.

[adrianh]

I got to thinking about this kernel panic... It is unlikely that the firmware is borked - if that were the case, I doubt that it would even get out of the gate. Now the kernel panic in this case is the result of the kernel finding a free page (i.e., the valid bit in the PTE is clear) where it thinks it should have a valid page. Assuming that the kernel is fine (the scope did work once after all), that means that when the kernel wrote the PTE to set the page base along with the valid and R/W/X bits, the valid bit (and maybe others as well) did not get set. So bad DDR DRAM.

So I bought four new DRAMs ($2.50 each at mouser). The plan was to desolder and replace each dram in turn, verifying that the scope didn't break worse after each one.

Well, I got lucky. The first DRAM I replaced fixed the problem, all the POST passed and the scope is functional!

Now all I need to do is find a replacement map/zoom button, get some probes, and get it calibrated...

[james_s]

Wow that's great, thanks for posting the update. Always feels good to fix something like that.

[pcwrangler]

Great job! Thank you for updating the thread.

Care to speculate on the bootloader freezing at the F in the FLASH: 64MB line? I haven't found the solution yet.

[adrianh]

Great job! Thank you for updating the thread.

Care to speculate on the bootloader freezing at the F in the FLASH: 64MB line? I haven't found the solution yet.

I've been thinking about your failure... So my failure was caused by bad DRAM - DRAM that was once good. In a way - I was very lucky because I got a kernel fault with bad page tables which clearly indicated that a location had gone bad. But bad dram in kernel space could have many other symptoms. For example, let's assume the OS is unpacked from a ROM to DRAM. The kernel is happily going along and then gets an interrupt (timer, device, whatever) and the ISR has some corrupt instructions.. An infinite exception loop could result.

I'm obviously speculating, but clearly bad DRAM is (now) a problem with these scopes. The DRAMs are cheap and the repair is fairly straightforward, so I'd say start replacing your DRAM (start with the bottom one on the CPU side) and see how it goes. Worst you will lose is a few hours and ~$20.

[pcwrangler]

Thanks Adrian, that gives me a little hope. I don't think it is even loading the kernel yet because uBoot doesn't get to that point. I just don't know enough about uBoot to determine if it's interacting with DRAM and would fail if DRAM was bad. Either way, the low cost of replacement is worthwhile.

[adrianh]

Can you post a link to your boot log? I'm wondering if I could find the generic uboot source given the version (or maybe circa the kernel version year) and figure out what it is doing. It would've been modified heavily by Tek but the overall arc of what it is doing might be helpful. At some point it will unpack the kernel from Somewhere into DRAM, but when is the key.

[pcwrangler]

This is the entirety of the boot log. It stops at the same place every time. If you want to read more, I started a thread a while back here: https://www.eevblog.com/forum/repair/tek-mso4104-no-boot/msg3596092/#msg3596092

Code: [Select]

U-Boot 1.1.4 (Jan  8 2007 - 11:12:14) Tektronix, Inc. V1.06

CPU:   AMCC PowerPC 440EP Rev. C at 333.333 MHz (PLB=133, OPB=66, EBC=66 MHz)
       I2C boot EEPROM enabled                                               
       Internal PCI arbiter enabled, PCI async ext clock used               
       32 kB I-Cache 32 kB D-Cache                                           
Board: Tektronix Route66 IBM 440EP Main Board                               
        VCO: 666 MHz                                                         
        CPU: 333 MHz                                                         
        PLB: 133 MHz                                                         
        OPB: 66 MHz
        EPB: 66 MHz
I2C:   ready
DRAM:  128 MB
F

I appreciate any insight you might have. The more input the better I can wrap my head around things. 

[adrianh]

I'm still thinking a problem with DRAM. This is why: As soon as uboot has figured out how much RAM is in the system (it might be hardcoded either in source or some blob somewhere) it is going to point its stack somewhere in the DRAM area so that it can actually start doing more complex activities. If the stack is pointing to bad DRAM then the system is going to get completely wedged - which certainly looks like your case.

[pcwrangler]

That makes sense and is at least worth a try. I ordered the last 3 Digikey had in stock. Wish me luck.

[adrianh]

Good luck! And if you need one or two extra DRAMS, drop me a PM. FWIW, I used air to get the old chips off the board (and lead-free takes some HOT air), cleaned the pads, and then drag soldered with lots of flux. I put kapton tape on all the little guys nearby and I didn't lose anyone!

[pcwrangler]

I appreciate the offer and will definitely let you know. That's the same technique I usually do with one additional step; I add leaded solder to the leads to lower the temp required to remove the chips with hot air. That way I don't stress the board and hopefully don't kill the chips just in case they are still good. Still, a little stressful working on a valuable piece of equipment. I'll let you know how it goes.

[pcwrangler]

Update: I just replaced both DRAM chips on the CPU side and it behaves the same exact way. I supposed that is good news that I didn't make it worse but it's still non-op. I hesitate to continue after seeing no progress. What do you think?

[adrianh]

Well… I think you should keep going. The stack would be in high mem so maybe the other side of the board(?). And to motivate you I’ll donate two drams to the cause. PM me.

[aibi1590]

Hey I got one mso2012 and a white screen appears.
I watch the ubootlog and stop at DRAM:64MB
I infer that the flash is faulty.

[adrianh]

That looks basically identical to pcwrangler's failure. I'm curious how his DRAM experiment will turn out...

[aibi1590]

That looks basically identical to pcwrangler's failure. I'm curious how his DRAM experiment will turn out...

Yes, but I don't know if it is DRAM or flash failure.
The four DRAMs on my MSO2012 are not the same.
And I don’t think I can buy a new flash and pre-burn the firmware into it.

[pcwrangler]

SUCCESS! I replaced the DRAM and it came right up. I can't thank Adrian enough for the support on this. Although only 2 confirmed cases is anecdotal so far, Adrian might be right; these scopes could have faulty DRAM. I hope more people see this thread.

aibi1590, your bootlog stops almost exactly where mine did. Even though your version is 2000 series, I would still bet the RAM is bad. Give it a try, the cost is minimal.


Now...... where did I put all those screws?!

[pcwrangler]

For the record: The bad ram was the bottom one on the back side (opposite of CPU) of the board.

[adrianh]

SWEET! You can ultimately thank the early linux mmu developers for being professional programmers and asserting what should be true. I work in CPU development so I knew what was going on (we boot linux all the time on our simulators and real hardware). Hopefully more folks can get these amazing scopes working again!

[aibi1590]

OK!I will try it.
very grateful to adrianh and pcwrangler.
I have ordered the new DRAM in a few weeks and I will report the result.

[aibi1590]

Eventually I replaced all the RAM but the problem remained the same

[maxwelllls]

Eventually I replaced all the RAM but the problem remained the same

You can try to measure the impedance of each memory data line to GND, by comparing the parameters of multiple chips to find abnormal values of pins, perhaps some CPU pins are rosin joint

[MarkF3837]

Thank you, community, for sleuthing the DPO4xxx 'scopes. I'm thinking (hoping) my DPO4104 might also have bad RAM. This scope also has a "stuck in splash screen" fault. Before I knew about the debug serial port, I tried updating firmware via a USB stick. The update started, but got stuck somewhere. Now that I have a debug port dump, I can see where it fails, but I don't "speak uBoot", so I'm not sure what to make of it.

Erasing and flashing flash pages goes along just fine, until it attempts a file mount. A snapshot of the key part of the dump follows:

    (boot & update messages)
    ...
Erasing 128 Kibyte @ 160000 -- 91 % complete.
  - Writing to flash; this may take a while...
Finished updating backup kernel.
Updating filesystem... DO NOT TURN OFF THE SYSTEM!!!!
Extracting the contents of /usr/local/perm...
  - Erasing flash... this may take a while.
Erasing 128 Kibyte @ 1de0000 -- 99 % complete.
mount: Mounting /dev/mtdblock7 on /mnt/rootfs failed: Invalid argument
  - Writing to flash; this may take a while...
tar: Cannot create directory `./usr/local': No space left on device
tar: Cannot create directory `./usr/local': No space left on device
    ...
    (repeat a bunch of times)
    ...
tar: Cannot create directory `./usr/local/': No space left on device
tar: ./usr/local/bin/scopeApp.ppcep: No such file or directory
An error occured while updating the root filesystem!
    (the end)

Looks like tar was uncompressing a file, but had nowhere to go.
Best case scenario, I replace some RAM and all is good. Worst case -- it's bricked.

-Mark