Keysight MSOX3104T Start-Up Problem!

[TheSteve]

Assuming it is a NAND corruption issue(likely) then hopefully Keysight will fix it for free. If they won't do it free then I am 99% sure I can help you recover it.

[GlowingGhoul]

Assuming it is a NAND corruption issue(likely) then hopefully Keysight will fix it for free. If they won't do it free then I am 99% sure I can help you recover it.

I appreciate that. If they come back with a large repair quote, as I expect, I'll still have to pay $311 to get the scope back. Hopefully I can get it running again so I can sell it and move on to another brand.

[bicycleguy]

I'll take that nasty MSOX3104T off your hands.

How much?

[tautech]

Assuming it is a NAND corruption issue(likely) then hopefully Keysight will fix it for free. If they won't do it free then I am 99% sure I can help you recover it.

I appreciate that. If they come back with a large repair quote, as I expect, I'll still have to pay $311 to get the scope back. Hopefully I can get it running again so I can sell it and move on to another brand.

You just might want to follow this:
https://www.eevblog.com/forum/testgear/test-equipment-anonymous-(tea)-group-therapy-thread/msg1830599/#msg1830599
https://www.eevblog.com/forum/testgear/test-equipment-anonymous-(tea)-group-therapy-thread/msg1887386/#msg1887386
https://www.eevblog.com/forum/testgear/test-equipment-anonymous-(tea)-group-therapy-thread/msg1894256/#msg1894256
https://www.eevblog.com/forum/testgear/test-equipment-anonymous-(tea)-group-therapy-thread/msg1897616/#msg1897616

Yes you read it right, member URI bought a busted Agilent DSO-X 3034A and is gunna get it repaired free by KS !
You need contact Daniel to get this sorted if the call center staff have no idea of what's going on:
https://www.eevblog.com/forum/profile/?u=114509

[GlowingGhoul]

I see they did. However, pointing that out got me nowhere.

$311 for a diagnosis, then $XXXX for the repair.

I guess my mistake was buying a brand new one from an authorized reseller for $5k, rather than buying a broken boat anchor for $100 from eBay
.

[kcbrown]

As an embedded engineer, if it were me designing a piece of equipment like this I would have implemented a fail-safe way to update the FW even if the FLASH gets corrupted and the unit won't boot. 

They did, in later version of firmware...

Of course, that doesn't help you if the fail-safe part of the NAND also gets corrupted ...

I've always thought that the best way to do these things is to have a known good version (i.e., the version that was available at the time of manufacture) of the firmware in ROM that one can boot from in the event the NAND image gets toasted somehow.  Not only would that provide an alternate boot in the event that the firmware upgrade fails, it provides a hardware-level failsafe in the event the NAND chip itself fails.

[nctnico]

As an embedded engineer, if it were me designing a piece of equipment like this I would have implemented a fail-safe way to update the FW even if the FLASH gets corrupted and the unit won't boot. 

They did, in later version of firmware...

Of course, that doesn't help you if the fail-safe part of the NAND also gets corrupted ...

It is more likely the original error correction isn't good enough to catch a few bit flips.

[GlowingGhoul]

I've spoken to 3 different people at Keysight regarding this, they all consistently claim:

1) There is no service note regarding NAND corruption or repairs out of warranty
2) There is no 'known issue' regarding NAND/firmware corruption
3) There is no coverage for any known issue beyond the warranty period. Either pay the $311 diagnosis fee+repair parts and labor cost, or it will not be repaired.

[Fgrir]

This thread has a lot of the original discussions, and interestingly Daniel at the time pointed out that the out-of-warranty repair was not a forever thing:

I should say (before it's engraved in stone somewhere) that this no-boot free repair isn't necessarily going to be around forever and ever (we have no plans to remove it) and IT DOESN'T APPLY TO EVERY SCOPE MODEL! Right now, it's just for the 2k, 3k, and 4k scopes that were shipped prior to 2016.

Funny thing is as I was re-reading it I saw GlowingGhoul was well aware of the issue and not so full of sympathy:

bgsd isn't entitled to anything. Keysight is living up to the terms of his warranty. For a little extra he could have gotten the insurance that would have covered him outside of the warranty but didn't.

That's not Keysight's fault. They owe him nothing.

Karma is a bitch...

[GlowingGhoul]

Funny thing is as I was re-reading it I saw GlowingGhoul was well aware of the issue and not so full of sympathy:

bgsd isn't entitled to anything. Keysight is living up to the terms of his warranty. For a little extra he could have gotten the insurance that would have covered him outside of the warranty but didn't.

That's not Keysight's fault. They owe him nothing.

Karma is a bitch...

Tell me, before I demonstrate what an intellectually dishonest fool you are, what was the fault I was referring to? Was it a known and widespread design defect that triggered by the innocuous behavior of leaving the scope off for a time? Well? Go ahead and tell everyone what sort of out-of-warranty fault I was referring to.

[GlowingGhoul]

I'll fill in the details. Keysight replaced a defective unit (for the corrupt nand issue) with a new one at no charge. User said he didn't try the replacement unit for 6 months after receiving it. When he did try it, it had a PLL error, which Keysight would not fix for free.

THAT is what my comment was in response to, in context.

[Fgrir]

Admittedly his situation was more complex than yours, but I'll stand by the comparison to someone with an out-of warranty situation who didn't choose to pay the "insurance" of the extended service contract.

I'm not trying to tear you down, and I'm sorry that you took it that way.  I really do hope you get your scope fixed for free.

[GlowingGhoul]

Admittedly his situation was more complex than yours, but I'll stand by the comparison to someone with an out-of warranty situation who didn't choose to pay the "insurance" of the extended service contract.

I'm not trying to tear you down, and I'm sorry that you took it that way.  I really do hope you get your scope fixed for free.

I would not be on here complaining about them not fixing an out of warranty failure that didn't involve a multi-year, thoroughly known and documented design flaw. The proper thing to do here is to acknowledge the problem and offer free extended coverage on that fault alone, instead of trying to cover it up by providing limited free repairs while giving themselves the ability to absolve themselves  of responsibility when it suits them.

[kcbrown]

Of course, that doesn't help you if the fail-safe part of the NAND also gets corrupted ...

It is more likely the original error correction isn't good enough to catch a few bit flips.

No doubt.   But with sufficiently bad luck, the area which houses the previous version of the firmware may also be corrupted by the same mechanism.

I suppose with sufficient ECC bits per data word, you'll drop the probability of an uncorrectable error low enough that it no longer matters.  But either your ECC mechanism is stored in the flash itself, in which case you've got a bootstrapping problem (how do you deal with the situation where the ECC code itself gets corrupted?), or it's in hardware/ROM.  It sounds like, in Keysight's case, it might live in the flash itself (though it may be that the measures they've taken involve periodic scans of the flash to look for errors and to actively rewrite the afflicted pages).   So you really do want a hardened alternate boot mechanism that will at least make it possible to restore the flash contents to sanity.  And if you're going to go to that trouble, you may as well make that mechanism be a ROM with a known-good image of your firmware, so that you're guaranteed to at least be able to use the scope even if the flash is so compromised that it is no longer writable.

[rsjsouza]

This comparison is still valid, and quite ironic considering the vitriolic tone of your words then. 

Admittedly his situation was more complex than yours, but I'll stand by the comparison to someone with an out-of warranty situation who didn't choose to pay the "insurance" of the extended service contract.

I don't need an imagination to analyze this situation.

Your equipment failed many months after the warranty expired. You believe you're entitled to a free repair or replacement despite the clear warranty terms and the ability to buy an inexpensive extended warranty. You took a chance and lost, now you cry like a spoiled brat.

Instead of taking the lesson of your stupidity, you want to make Keysight the bad actor.

[GlowingGhoul]

My 'tone' makes the comparison valid? You see no difference between a company acting responsibly to address a serious design flaw, and a company declining to repair a non-design flaw related failure out of warranty?

Apparently my 'tone' has driven you to disregard logic out of a desire to criticize me because I offended your delicate sensibilities.

[Pinkus]

Right now, it's just for the 2k, 3k, and 4k scopes that were shipped prior to 2016.

As my DSOX4000 scope starts to often need multiple on/off switches to start up (it often hangs during boot with only the Ref LED illuminated) I am afraid of an upcoming NAND problem.
(btw: I would appreciate any ideas how I might minimize any future/potential NAND problem now; would a backup help?)

As Daniel explicitly says "prior 2016": two questions are coming up for me:
1) What was changed after 2015? New NANDs (or different manufacturer)?
2) I assume the NAND is a separate chip. Would it be possible to desolder and change the NAND against a new one?

[GlowingGhoul]

Upgrade to the latest firmware. They've been incorporating measures to minimize the chances of corruption preventing a boot.

[Sal Ammoniac]

Just spent another hour attempting to get it to boot without luck. I'll be sending it back tomorrow.

Perhaps I should ask them to replace the on/off switch too--it's gotten quite a workout in the past few days. 

[Pinkus]

Upgrade to the latest firmware. They've been incorporating measures to minimize the chances of corruption preventing a boot.

I've already been using the latest firmware for a year. Since DSOX3000T are also affected, the newer firmware obviously is not a miracle cure. Possibly a time bomb is sleeping in all DSOX scopes. Hence my question about the possibility to exchange the NAND chip.

[Keysight DanielBogdanoff]

PM sent. I'll try to get you fixed up.

Our service centers are excellent 99.9% of the time, but every once in a while something slips through. I'm happy when I can catch it and help here.

We've done extensive work on making the NAND more robust, I highly recommend that everyone update their firmware. This includes a better backup scheme and a recovery image when there are issues detected. Old firmware won't have this, new firmware does.

I'll also say that if your scope is out of warranty, the service centers will almost always quote a charge to inspect/repair. If it turns out to be a NAND issue, you won't get charged. But, they can't quote free because they don't know for sure that it is NAND. At least, that's how it was explained to me.

I'll get info from you through PMs and try to work some magic on the back end.

-Daniel

[Sal Ammoniac]

I got my scope back from Keysight yesterday and it now boots normally.

The repair turn-around wasn't too bad. I shipped the scope on October 22nd and got it back on October 30th, so a little over a week door-to-door.

The service order details said they replaced the acquisition system board. This was more than I expected was needed here. I was expecting them to update the flash to the latest firmware revision using an internal JTAG or debug header.

This whole episode leaves a bad taste in my mouth. I'm always going to wonder if the scope is going to boot up every time I need to use it. Based on this experience, my next scope probably won't be a Keysight.

[Keysight DanielBogdanoff]

Hi Sal,

Our standard process is to swap the board out, that cuts down on the very expensive repair time. I've talked through it more on a couple other threads here, but troubleshooting and repair time can easily exceed the price of an InfiniiVision scope, so we just do a swap and refurb the board at the factory. As with all electronics, we can't guarantee that it'll be trouble free forever. But, the new NAND protection in our firmware is orders of magnitude more reliable than previous versions (internal data backs up that claim).

Again, I'm sorry for the trouble you had and the time you spent troubleshooting. If you have any issues, feel free to PM me and we'll get you taken care of!

[sibeen]

Every time this, or a similar thread, comes up in the list of active threads I turn around and turn on my 3024T

[Sal Ammoniac]

As with all electronics, we can't guarantee that it'll be trouble free forever. But, the new NAND protection in our firmware is orders of magnitude more reliable than previous versions (internal data backs up that claim).

I wonder how the original design made it through the design review process? This is really a rhetorical question as I wouldn't expect you to know.

I bought this scope directly from Keysight, so I'm also wondering why they didn't notify me via email (they had my address from the order) to update the FW? I've gotten written notices from car companies every time they have a recall (and some of these were merely annoying issues, not just safety-related issues)--perhaps Keysight should do something similar.