Keysight MSOX3104T Start-Up Problem!

[Sal Ammoniac]

My backup oscilloscope, a Keysight MSOX3104T, doesn't complete its start-up sequence and appears dead.

This is pretty much a brand new scope. I've had it for two years, but it's gotten very little use since it's my backup scope. It's been unplugged from the wall outlet for over a year. When I plugged it back in today and pushed the power button, it turns on the LED next to the [Ref] key and then turns on the LED next to the [Math]key, but it stops there and doesn't get any further. The screen itself is completely black.

I can't try updating the FW because it doesn't get far enough to allow me to attempt that.

Anyone seen this before? Is there a fix? 

[HighVoltage]

It is probably a NAND corruption and Keysight will fix it under warranty, even if the warranty has expired.

[GlowingGhoul]

EXECTLY the same thing occurred to me today, except my scope was last used 3 months ago. Please let me know how Keysight handles this for you.

[Sal Ammoniac]

EXECTLY the same thing occurred to me today, except my scope was last used 3 months ago. Please let me know how Keysight handles this for you.

I plan on calling Keysight tomorrow. I'll let you know how it goes.

I've found a number of other descriptions of this happening to people with MSOX3000 and MSOX4000 series scopes by doing a Google search. The common link between the incidents appears to be not using the scope for a period of several months. Why this should cause the scopes to not power up properly is beyond me. Perhaps there's some CMOS configuration memory that's kept alive by a rechargeable battery or supercap that discharges if the scope isn't used for a long period of time?

[sibeen]

Rushes over and turns on my 3024T.

I probably haven't had it on for a month or so.

[TheSteve]

EXECTLY the same thing occurred to me today, except my scope was last used 3 months ago. Please let me know how Keysight handles this for you.

I plan on calling Keysight tomorrow. I'll let you know how it goes.

I've found a number of other descriptions of this happening to people with MSOX3000 and MSOX4000 series scopes by doing a Google search. The common link between the incidents appears to be not using the scope for a period of several months. Why this should cause the scopes to not power up properly is beyond me. Perhaps there's some CMOS configuration memory that's kept alive by a rechargeable battery or supercap that discharges if the scope isn't used for a long period of time?

Seems to just be a bit or two in the NAND flash memory that decides to alter itself. Older firmware doesn't recover from the issue very well, newer firmware does. Scopes have a 3 year warranty and Keysight is pretty good with warranty repairs.

[Sal Ammoniac]

EXECTLY the same thing occurred to me today, except my scope was last used 3 months ago. Please let me know how Keysight handles this for you.

I plan on calling Keysight tomorrow. I'll let you know how it goes.

I've found a number of other descriptions of this happening to people with MSOX3000 and MSOX4000 series scopes by doing a Google search. The common link between the incidents appears to be not using the scope for a period of several months. Why this should cause the scopes to not power up properly is beyond me. Perhaps there's some CMOS configuration memory that's kept alive by a rechargeable battery or supercap that discharges if the scope isn't used for a long period of time?

Seems to just be a bit or two in the NAND flash memory that decides to alter itself. Older firmware doesn't recover from the issue very well, newer firmware does. Scopes have a 3 year warranty and Keysight is pretty good with warranty repairs.

I wonder if there’s a way to force a firmware update if something like this happens?

[TheSteve]

EXECTLY the same thing occurred to me today, except my scope was last used 3 months ago. Please let me know how Keysight handles this for you.

I plan on calling Keysight tomorrow. I'll let you know how it goes.

I've found a number of other descriptions of this happening to people with MSOX3000 and MSOX4000 series scopes by doing a Google search. The common link between the incidents appears to be not using the scope for a period of several months. Why this should cause the scopes to not power up properly is beyond me. Perhaps there's some CMOS configuration memory that's kept alive by a rechargeable battery or supercap that discharges if the scope isn't used for a long period of time?

Seems to just be a bit or two in the NAND flash memory that decides to alter itself. Older firmware doesn't recover from the issue very well, newer firmware does. Scopes have a 3 year warranty and Keysight is pretty good with warranty repairs.

I wonder if there’s a way to force a firmware update if something like this happens?

It can most likely be recovered if you have a network card by booting an image via network and then updating firmware. This has been covered in the 2000/3000 hacking thread for the most part. I've never done it with a T series scope, only the 2000a/3000a.

[Sal Ammoniac]

I wonder if there’s a way to force a firmware update if something like this happens?

It can most likely be recovered if you have a network card by booting an image via network and then updating firmware. This has been covered in the 2000/3000 hacking thread for the most part. I've never done it with a T series scope, only the 2000a/3000a.

Unfortunately, I don't have a network card. I was hoping for something simpler, like "Insert a USB drive with new FW in the USB slot and power-up the unit while holding down the X, Y, and Z keys".

[Sal Ammoniac]

EXECTLY the same thing occurred to me today, except my scope was last used 3 months ago. Please let me know how Keysight handles this for you.

I spoke to Keysight tech support this morning. They're very friendly and knowledgeable. He acknowledged that this is a known problem that affects these scopes and Keysight will fix it even if the scope is no longer under warranty.

As HighVoltage and TheSteve said, it seems that this is a FLASH corruption issue that earlier versions of FW couldn't recover from. 

The tech said that if you can get it to complete the start-up sequence just once, you can load the latest FW and that will fix the problem. I plan to spend an hour or so tonight trying to get it to boot before I box it up and ship it back to Keysight for repair.

As an embedded engineer, if it were me designing a piece of equipment like this I would have implemented a fail-safe way to update the FW even if the FLASH gets corrupted and the unit won't boot. 

[2N3055]

As an embedded engineer, if it were me designing a piece of equipment like this I would have implemented a fail-safe way to update the FW even if the FLASH gets corrupted and the unit won't boot. 

They did, in later version of firmware...

[GlowingGhoul]

I wasn't able to speak with a tech. They took my information and said I should hear from someone within 24hours. They had no idea what I was referring to regarding NAND corruption issues.

My warranty ended 2 months ago, and I don't have a good feeling about this.

[HighVoltage]

My 3000X scope was almost 2 years out of warranty and Keysight fixed it for free, after I had send them the scope.

[Sal Ammoniac]

As an embedded engineer, if it were me designing a piece of equipment like this I would have implemented a fail-safe way to update the FW even if the FLASH gets corrupted and the unit won't boot. 

They did, in later version of firmware...

I'm talking about a hardware fail-safe here. A FW fix obviously doesn't help if there's no way to update the FW!

[2N3055]

As an embedded engineer, if it were me designing a piece of equipment like this I would have implemented a fail-safe way to update the FW even if the FLASH gets corrupted and the unit won't boot. 

They did, in later version of firmware...

I'm talking about a hardware fail-safe here. A FW fix obviously doesn't help if there's no way to update the FW!

Well I guess they know how... I guess they setup secondary image or ECC or something. It can detect firmware corruption and starts backup boot procedure...

[TheSteve]

The bootloader is in NOR flash, it loads the primary image from the NAND flash, if that fails it goes to a second image - in newer firmware anyway. So most of the time a network boot can recover the scope, unless some of the truly unique info was lost - model/serial, cal data, licenses etc. The newer firmware also handles remapping bad blocks so that both images are correct/clean.

[GlowingGhoul]

The bootloader is in NOR flash, it loads the primary image from the NAND flash, if that fails it goes to a second image - in newer firmware anyway. So most of the time a network boot can recover the scope, unless some of the truly unique info was lost - model/serial, cal data, licenses etc. The newer firmware also handles remapping bad blocks so that both images are correct/clean.

I'm a bit confused. Are you saying that you need to network boot vs normal boot to recover, or that recovery happens automatically, in which case what is the relevance of network boot? I have ethernet on this scope.

[TheSteve]

The bootloader is in NOR flash, it loads the primary image from the NAND flash, if that fails it goes to a second image - in newer firmware anyway. So most of the time a network boot can recover the scope, unless some of the truly unique info was lost - model/serial, cal data, licenses etc. The newer firmware also handles remapping bad blocks so that both images are correct/clean.

I'm a bit confused. Are you saying that you need to network boot vs normal boot to recover, or that recovery happens automatically, in which case what is the relevance of network boot? I have ethernet on this scope.

You'd need to open the scope and connect to the serial debug header. From there you can interact with the bootloader and tell it to tftp a prepared image via a network connection.

[Sal Ammoniac]

Spent 45 minutes last night repeatedly pushing the power button. No joy. I'll try the same tonight and if still no joy will pack it up and ship back to Keysight tomorrow morning. They quoted a 12-14 day turn-around time on the repair.

[GlowingGhoul]

It's been over 24 hours since I called Keysight and was told I'd be called back within that time. Not good.

[GlowingGhoul]

Just a quick update on Keysight's customer service. I just called, since they never called me back.

Bottom line, they will not assist, period. I'm on my own, 60 days out of warranty.

They denied knowing anything about 'corrupt nand', said there is no service note, and there is no 'known defect' of that type with any model.

When I asked about paid repair options, they demanded to see my origianl invoice, depsite purchasing the scope from Newark, and registering it with them the day it arrived, and purchasing numerous options that Keysight had to process.

Never again. A $5000 paperweight they refuse to repair. For what it's worth, don't tell them you are an individual, you MUST be a company if you want decent customer service. The moment I told them I had no company name, this was my personal scope, their attitude immediately changed from eager to help to suspicion. What the hell happened to Keysight?

[JonM]

Just a quick update on Keysight's customer service. I just called, since they never called me back.

Bottom line, they will not assist, period. I'm on my own, 60 days out of warranty.

They denied knowing anything about 'corrupt nand', said there is no service note, and there is no 'known defect' of that type with any model.

When I asked about paid repair options, they demanded to see my origianl invoice, depsite purchasing the scope from Newark, and registering it with them the day it arrived, and purchasing numerous options that Keysight had to process.

Never again. A $5000 paperweight they refuse to repair. For what it's worth, don't tell them you are an individual, you MUST be a company if you want decent customer service. The moment I told them I had no company name, this was my personal scope, their attitude immediately changed from eager to help to suspicion. What the hell happened to Keysight?

I am fairly certain that Keysight will fix this, but it might not get moving until Daniel Bogdanoff sees this thread. There have been multiple posts about the NAND corruption and it seems that Keysight fixed them all, even if past warranty period.

I just powered up my personally owned DSOX3024T to be sure that it is OK after sitting for a while, but I have recent firmware so it should not be an issue. I do wonder if I should get a service contract when my warranty runs out in another year. However, it's probably expensive for a lightly used, personally owned, non-profit instrument.

[GlowingGhoul]

I called back, livid. Keysight agreed to evaluate the unit for repair for a fee of $311, but I won't have to pay that until they give me the full quote for the repair (how generous), somewhere around $2000.

I was told there is no service note, no known widespread problem, no NAND fault repairs out of warranty.

[sibeen]

GlowingGhoul, perhaps you should mention where you are based. Someone, probably Daniel, from Keysight will eventually see this or be pointed to it. They'd need to know where you are from so the arse kicking can commence.

[GlowingGhoul]

GlowingGhoul, perhaps you should mention where you are based. Someone, probably Daniel, from Keysight will eventually see this or be pointed to it. They'd need to know where you are from so the arse kicking can commence.

I'm in the US, and was dealing with customer 'service' at Keysight HQ.

[TheSteve]

Assuming it is a NAND corruption issue(likely) then hopefully Keysight will fix it for free. If they won't do it free then I am 99% sure I can help you recover it.

[GlowingGhoul]

Assuming it is a NAND corruption issue(likely) then hopefully Keysight will fix it for free. If they won't do it free then I am 99% sure I can help you recover it.

I appreciate that. If they come back with a large repair quote, as I expect, I'll still have to pay $311 to get the scope back. Hopefully I can get it running again so I can sell it and move on to another brand.

[bicycleguy]

I'll take that nasty MSOX3104T off your hands.

How much?

[tautech]

Assuming it is a NAND corruption issue(likely) then hopefully Keysight will fix it for free. If they won't do it free then I am 99% sure I can help you recover it.

I appreciate that. If they come back with a large repair quote, as I expect, I'll still have to pay $311 to get the scope back. Hopefully I can get it running again so I can sell it and move on to another brand.

You just might want to follow this:
https://www.eevblog.com/forum/testgear/test-equipment-anonymous-(tea)-group-therapy-thread/msg1830599/#msg1830599
https://www.eevblog.com/forum/testgear/test-equipment-anonymous-(tea)-group-therapy-thread/msg1887386/#msg1887386
https://www.eevblog.com/forum/testgear/test-equipment-anonymous-(tea)-group-therapy-thread/msg1894256/#msg1894256
https://www.eevblog.com/forum/testgear/test-equipment-anonymous-(tea)-group-therapy-thread/msg1897616/#msg1897616

Yes you read it right, member URI bought a busted Agilent DSO-X 3034A and is gunna get it repaired free by KS !
You need contact Daniel to get this sorted if the call center staff have no idea of what's going on:
https://www.eevblog.com/forum/profile/?u=114509

[GlowingGhoul]

I see they did. However, pointing that out got me nowhere.

$311 for a diagnosis, then $XXXX for the repair.

I guess my mistake was buying a brand new one from an authorized reseller for $5k, rather than buying a broken boat anchor for $100 from eBay
.

[kcbrown]

As an embedded engineer, if it were me designing a piece of equipment like this I would have implemented a fail-safe way to update the FW even if the FLASH gets corrupted and the unit won't boot. 

They did, in later version of firmware...

Of course, that doesn't help you if the fail-safe part of the NAND also gets corrupted ...

I've always thought that the best way to do these things is to have a known good version (i.e., the version that was available at the time of manufacture) of the firmware in ROM that one can boot from in the event the NAND image gets toasted somehow.  Not only would that provide an alternate boot in the event that the firmware upgrade fails, it provides a hardware-level failsafe in the event the NAND chip itself fails.

[nctnico]

As an embedded engineer, if it were me designing a piece of equipment like this I would have implemented a fail-safe way to update the FW even if the FLASH gets corrupted and the unit won't boot. 

They did, in later version of firmware...

Of course, that doesn't help you if the fail-safe part of the NAND also gets corrupted ...

It is more likely the original error correction isn't good enough to catch a few bit flips.

[GlowingGhoul]

I've spoken to 3 different people at Keysight regarding this, they all consistently claim:

1) There is no service note regarding NAND corruption or repairs out of warranty
2) There is no 'known issue' regarding NAND/firmware corruption
3) There is no coverage for any known issue beyond the warranty period. Either pay the $311 diagnosis fee+repair parts and labor cost, or it will not be repaired.

[Fgrir]

This thread has a lot of the original discussions, and interestingly Daniel at the time pointed out that the out-of-warranty repair was not a forever thing:

I should say (before it's engraved in stone somewhere) that this no-boot free repair isn't necessarily going to be around forever and ever (we have no plans to remove it) and IT DOESN'T APPLY TO EVERY SCOPE MODEL! Right now, it's just for the 2k, 3k, and 4k scopes that were shipped prior to 2016.

Funny thing is as I was re-reading it I saw GlowingGhoul was well aware of the issue and not so full of sympathy:

bgsd isn't entitled to anything. Keysight is living up to the terms of his warranty. For a little extra he could have gotten the insurance that would have covered him outside of the warranty but didn't.

That's not Keysight's fault. They owe him nothing.

Karma is a bitch...

[GlowingGhoul]

Funny thing is as I was re-reading it I saw GlowingGhoul was well aware of the issue and not so full of sympathy:

bgsd isn't entitled to anything. Keysight is living up to the terms of his warranty. For a little extra he could have gotten the insurance that would have covered him outside of the warranty but didn't.

That's not Keysight's fault. They owe him nothing.

Karma is a bitch...

Tell me, before I demonstrate what an intellectually dishonest fool you are, what was the fault I was referring to? Was it a known and widespread design defect that triggered by the innocuous behavior of leaving the scope off for a time? Well? Go ahead and tell everyone what sort of out-of-warranty fault I was referring to.

[GlowingGhoul]

I'll fill in the details. Keysight replaced a defective unit (for the corrupt nand issue) with a new one at no charge. User said he didn't try the replacement unit for 6 months after receiving it. When he did try it, it had a PLL error, which Keysight would not fix for free.

THAT is what my comment was in response to, in context.

[Fgrir]

Admittedly his situation was more complex than yours, but I'll stand by the comparison to someone with an out-of warranty situation who didn't choose to pay the "insurance" of the extended service contract.

I'm not trying to tear you down, and I'm sorry that you took it that way.  I really do hope you get your scope fixed for free.

[GlowingGhoul]

Admittedly his situation was more complex than yours, but I'll stand by the comparison to someone with an out-of warranty situation who didn't choose to pay the "insurance" of the extended service contract.

I'm not trying to tear you down, and I'm sorry that you took it that way.  I really do hope you get your scope fixed for free.

I would not be on here complaining about them not fixing an out of warranty failure that didn't involve a multi-year, thoroughly known and documented design flaw. The proper thing to do here is to acknowledge the problem and offer free extended coverage on that fault alone, instead of trying to cover it up by providing limited free repairs while giving themselves the ability to absolve themselves  of responsibility when it suits them.

[kcbrown]

Of course, that doesn't help you if the fail-safe part of the NAND also gets corrupted ...

It is more likely the original error correction isn't good enough to catch a few bit flips.

No doubt.   But with sufficiently bad luck, the area which houses the previous version of the firmware may also be corrupted by the same mechanism.

I suppose with sufficient ECC bits per data word, you'll drop the probability of an uncorrectable error low enough that it no longer matters.  But either your ECC mechanism is stored in the flash itself, in which case you've got a bootstrapping problem (how do you deal with the situation where the ECC code itself gets corrupted?), or it's in hardware/ROM.  It sounds like, in Keysight's case, it might live in the flash itself (though it may be that the measures they've taken involve periodic scans of the flash to look for errors and to actively rewrite the afflicted pages).   So you really do want a hardened alternate boot mechanism that will at least make it possible to restore the flash contents to sanity.  And if you're going to go to that trouble, you may as well make that mechanism be a ROM with a known-good image of your firmware, so that you're guaranteed to at least be able to use the scope even if the flash is so compromised that it is no longer writable.

[rsjsouza]

This comparison is still valid, and quite ironic considering the vitriolic tone of your words then. 

Admittedly his situation was more complex than yours, but I'll stand by the comparison to someone with an out-of warranty situation who didn't choose to pay the "insurance" of the extended service contract.

I don't need an imagination to analyze this situation.

Your equipment failed many months after the warranty expired. You believe you're entitled to a free repair or replacement despite the clear warranty terms and the ability to buy an inexpensive extended warranty. You took a chance and lost, now you cry like a spoiled brat.

Instead of taking the lesson of your stupidity, you want to make Keysight the bad actor.

[GlowingGhoul]

My 'tone' makes the comparison valid? You see no difference between a company acting responsibly to address a serious design flaw, and a company declining to repair a non-design flaw related failure out of warranty?

Apparently my 'tone' has driven you to disregard logic out of a desire to criticize me because I offended your delicate sensibilities.

[Pinkus]

Right now, it's just for the 2k, 3k, and 4k scopes that were shipped prior to 2016.

As my DSOX4000 scope starts to often need multiple on/off switches to start up (it often hangs during boot with only the Ref LED illuminated) I am afraid of an upcoming NAND problem.
(btw: I would appreciate any ideas how I might minimize any future/potential NAND problem now; would a backup help?)

As Daniel explicitly says "prior 2016": two questions are coming up for me:
1) What was changed after 2015? New NANDs (or different manufacturer)?
2) I assume the NAND is a separate chip. Would it be possible to desolder and change the NAND against a new one?

[GlowingGhoul]

Upgrade to the latest firmware. They've been incorporating measures to minimize the chances of corruption preventing a boot.

[Sal Ammoniac]

Just spent another hour attempting to get it to boot without luck. I'll be sending it back tomorrow.

Perhaps I should ask them to replace the on/off switch too--it's gotten quite a workout in the past few days. 

[Pinkus]

Upgrade to the latest firmware. They've been incorporating measures to minimize the chances of corruption preventing a boot.

I've already been using the latest firmware for a year. Since DSOX3000T are also affected, the newer firmware obviously is not a miracle cure. Possibly a time bomb is sleeping in all DSOX scopes. Hence my question about the possibility to exchange the NAND chip.

[Keysight DanielBogdanoff]

PM sent. I'll try to get you fixed up.

Our service centers are excellent 99.9% of the time, but every once in a while something slips through. I'm happy when I can catch it and help here.

We've done extensive work on making the NAND more robust, I highly recommend that everyone update their firmware. This includes a better backup scheme and a recovery image when there are issues detected. Old firmware won't have this, new firmware does.

I'll also say that if your scope is out of warranty, the service centers will almost always quote a charge to inspect/repair. If it turns out to be a NAND issue, you won't get charged. But, they can't quote free because they don't know for sure that it is NAND. At least, that's how it was explained to me.

I'll get info from you through PMs and try to work some magic on the back end.

-Daniel

[Sal Ammoniac]

I got my scope back from Keysight yesterday and it now boots normally.

The repair turn-around wasn't too bad. I shipped the scope on October 22nd and got it back on October 30th, so a little over a week door-to-door.

The service order details said they replaced the acquisition system board. This was more than I expected was needed here. I was expecting them to update the flash to the latest firmware revision using an internal JTAG or debug header.

This whole episode leaves a bad taste in my mouth. I'm always going to wonder if the scope is going to boot up every time I need to use it. Based on this experience, my next scope probably won't be a Keysight.

[Keysight DanielBogdanoff]

Hi Sal,

Our standard process is to swap the board out, that cuts down on the very expensive repair time. I've talked through it more on a couple other threads here, but troubleshooting and repair time can easily exceed the price of an InfiniiVision scope, so we just do a swap and refurb the board at the factory. As with all electronics, we can't guarantee that it'll be trouble free forever. But, the new NAND protection in our firmware is orders of magnitude more reliable than previous versions (internal data backs up that claim).

Again, I'm sorry for the trouble you had and the time you spent troubleshooting. If you have any issues, feel free to PM me and we'll get you taken care of!

[sibeen]

Every time this, or a similar thread, comes up in the list of active threads I turn around and turn on my 3024T

[Sal Ammoniac]

As with all electronics, we can't guarantee that it'll be trouble free forever. But, the new NAND protection in our firmware is orders of magnitude more reliable than previous versions (internal data backs up that claim).

I wonder how the original design made it through the design review process? This is really a rhetorical question as I wouldn't expect you to know.

I bought this scope directly from Keysight, so I'm also wondering why they didn't notify me via email (they had my address from the order) to update the FW? I've gotten written notices from car companies every time they have a recall (and some of these were merely annoying issues, not just safety-related issues)--perhaps Keysight should do something similar.

[JonM]

As with all electronics, we can't guarantee that it'll be trouble free forever. But, the new NAND protection in our firmware is orders of magnitude more reliable than previous versions (internal data backs up that claim).

I wonder how the original design made it through the design review process? This is really a rhetorical question as I wouldn't expect you to know.

I bought this scope directly from Keysight, so I'm also wondering why they didn't notify me via email (they had my address from the order) to update the FW? I've gotten written notices from car companies every time they have a recall (and some of these were merely annoying issues, not just safety-related issues)--perhaps Keysight should do something similar.

I bought my scope through TestEquity but I have it registered under my MyKeysight account which does cause me to get some emails about the products I own. I believe that I have received notifications of new software for the scope but don't remember any that suggested it was an urgent update (I learned of the NAND issue on this forum). Checking one of the messages I see that I asked for certain updates:

Your current email subscription             Frequency   Selection
Forum update                                 Weekly   My assets only
Product Manual                               Weekly   My assets only
Application notes                            Weekly   My assets only
Software update alert                        Weekly   My assets only
New FAQs                                     Weekly   My assets only

I will note that I see a lot of forum updates for "Truevolt Series 6.5 & 7.5 Digit Multimeters" that have nothing to do with a DMM.