Author Topic: Keysight MSOX3104T Start-Up Problem!  (Read 5148 times)

0 Members and 1 Guest are viewing this topic.

Offline Sal Ammoniac

  • Super Contributor
  • ***
  • Posts: 1249
  • Country: us
Keysight MSOX3104T Start-Up Problem!
« on: October 14, 2018, 09:49:23 pm »
My backup oscilloscope, a Keysight MSOX3104T, doesn't complete its start-up sequence and appears dead.

This is pretty much a brand new scope. I've had it for two years, but it's gotten very little use since it's my backup scope. It's been unplugged from the wall outlet for over a year. When I plugged it back in today and pushed the power button, it turns on the LED next to the [Ref] key and then turns on the LED next to the [Math]key, but it stops there and doesn't get any further. The screen itself is completely black.

I can't try updating the FW because it doesn't get far enough to allow me to attempt that.

Anyone seen this before? Is there a fix?  |O
Complexity is the number-one enemy of high-quality code.
 

Online HighVoltage

  • Super Contributor
  • ***
  • Posts: 4602
  • Country: de
Re: Keysight MSOX3104T Start-Up Problem!
« Reply #1 on: October 14, 2018, 09:59:12 pm »
It is probably a NAND corruption and Keysight will fix it under warranty, even if the warranty has expired.
There are 3 kinds of people in this world, those who can count and those who can not.
 

Offline GlowingGhoul

  • Regular Contributor
  • *
  • Posts: 236
Re: Keysight MSOX3104T Start-Up Problem!
« Reply #2 on: October 14, 2018, 11:05:31 pm »
EXECTLY the same thing occurred to me today, except my scope was last used 3 months ago. Please let me know how Keysight handles this for you.
 

Offline Sal Ammoniac

  • Super Contributor
  • ***
  • Posts: 1249
  • Country: us
Re: Keysight MSOX3104T Start-Up Problem!
« Reply #3 on: October 15, 2018, 02:06:32 am »
EXECTLY the same thing occurred to me today, except my scope was last used 3 months ago. Please let me know how Keysight handles this for you.

I plan on calling Keysight tomorrow. I'll let you know how it goes.

I've found a number of other descriptions of this happening to people with MSOX3000 and MSOX4000 series scopes by doing a Google search. The common link between the incidents appears to be not using the scope for a period of several months. Why this should cause the scopes to not power up properly is beyond me. Perhaps there's some CMOS configuration memory that's kept alive by a rechargeable battery or supercap that discharges if the scope isn't used for a long period of time?
Complexity is the number-one enemy of high-quality code.
 

Offline sibeen

  • Frequent Contributor
  • **
  • Posts: 271
  • Country: au
Re: Keysight MSOX3104T Start-Up Problem!
« Reply #4 on: October 15, 2018, 02:13:49 am »
Rushes over and turns on my 3024T.

I probably haven't had it on for a month or so.
 

Online TheSteve

  • Supporter
  • ****
  • Posts: 3272
  • Country: ca
  • Living the Dream
Re: Keysight MSOX3104T Start-Up Problem!
« Reply #5 on: October 15, 2018, 02:18:16 am »
EXECTLY the same thing occurred to me today, except my scope was last used 3 months ago. Please let me know how Keysight handles this for you.

I plan on calling Keysight tomorrow. I'll let you know how it goes.

I've found a number of other descriptions of this happening to people with MSOX3000 and MSOX4000 series scopes by doing a Google search. The common link between the incidents appears to be not using the scope for a period of several months. Why this should cause the scopes to not power up properly is beyond me. Perhaps there's some CMOS configuration memory that's kept alive by a rechargeable battery or supercap that discharges if the scope isn't used for a long period of time?

Seems to just be a bit or two in the NAND flash memory that decides to alter itself. Older firmware doesn't recover from the issue very well, newer firmware does. Scopes have a 3 year warranty and Keysight is pretty good with warranty repairs.
VE7FM
 

Offline Sal Ammoniac

  • Super Contributor
  • ***
  • Posts: 1249
  • Country: us
Re: Keysight MSOX3104T Start-Up Problem!
« Reply #6 on: October 15, 2018, 04:00:11 am »
EXECTLY the same thing occurred to me today, except my scope was last used 3 months ago. Please let me know how Keysight handles this for you.

I plan on calling Keysight tomorrow. I'll let you know how it goes.

I've found a number of other descriptions of this happening to people with MSOX3000 and MSOX4000 series scopes by doing a Google search. The common link between the incidents appears to be not using the scope for a period of several months. Why this should cause the scopes to not power up properly is beyond me. Perhaps there's some CMOS configuration memory that's kept alive by a rechargeable battery or supercap that discharges if the scope isn't used for a long period of time?

Seems to just be a bit or two in the NAND flash memory that decides to alter itself. Older firmware doesn't recover from the issue very well, newer firmware does. Scopes have a 3 year warranty and Keysight is pretty good with warranty repairs.

I wonder if there’s a way to force a firmware update if something like this happens?
Complexity is the number-one enemy of high-quality code.
 

Online TheSteve

  • Supporter
  • ****
  • Posts: 3272
  • Country: ca
  • Living the Dream
Re: Keysight MSOX3104T Start-Up Problem!
« Reply #7 on: October 15, 2018, 04:21:44 am »
EXECTLY the same thing occurred to me today, except my scope was last used 3 months ago. Please let me know how Keysight handles this for you.

I plan on calling Keysight tomorrow. I'll let you know how it goes.

I've found a number of other descriptions of this happening to people with MSOX3000 and MSOX4000 series scopes by doing a Google search. The common link between the incidents appears to be not using the scope for a period of several months. Why this should cause the scopes to not power up properly is beyond me. Perhaps there's some CMOS configuration memory that's kept alive by a rechargeable battery or supercap that discharges if the scope isn't used for a long period of time?

Seems to just be a bit or two in the NAND flash memory that decides to alter itself. Older firmware doesn't recover from the issue very well, newer firmware does. Scopes have a 3 year warranty and Keysight is pretty good with warranty repairs.

I wonder if there’s a way to force a firmware update if something like this happens?

It can most likely be recovered if you have a network card by booting an image via network and then updating firmware. This has been covered in the 2000/3000 hacking thread for the most part. I've never done it with a T series scope, only the 2000a/3000a.
VE7FM
 

Offline Sal Ammoniac

  • Super Contributor
  • ***
  • Posts: 1249
  • Country: us
Re: Keysight MSOX3104T Start-Up Problem!
« Reply #8 on: October 15, 2018, 01:41:58 pm »
I wonder if there’s a way to force a firmware update if something like this happens?

It can most likely be recovered if you have a network card by booting an image via network and then updating firmware. This has been covered in the 2000/3000 hacking thread for the most part. I've never done it with a T series scope, only the 2000a/3000a.

Unfortunately, I don't have a network card. I was hoping for something simpler, like "Insert a USB drive with new FW in the USB slot and power-up the unit while holding down the X, Y, and Z keys".
Complexity is the number-one enemy of high-quality code.
 

Offline Sal Ammoniac

  • Super Contributor
  • ***
  • Posts: 1249
  • Country: us
Re: Keysight MSOX3104T Start-Up Problem!
« Reply #9 on: October 15, 2018, 04:53:45 pm »
EXECTLY the same thing occurred to me today, except my scope was last used 3 months ago. Please let me know how Keysight handles this for you.

I spoke to Keysight tech support this morning. They're very friendly and knowledgeable. He acknowledged that this is a known problem that affects these scopes and Keysight will fix it even if the scope is no longer under warranty.

As HighVoltage and TheSteve said, it seems that this is a FLASH corruption issue that earlier versions of FW couldn't recover from.  :-BROKE

The tech said that if you can get it to complete the start-up sequence just once, you can load the latest FW and that will fix the problem. I plan to spend an hour or so tonight trying to get it to boot before I box it up and ship it back to Keysight for repair.

As an embedded engineer, if it were me designing a piece of equipment like this I would have implemented a fail-safe way to update the FW even if the FLASH gets corrupted and the unit won't boot.  :-+
Complexity is the number-one enemy of high-quality code.
 

Offline 2N3055

  • Super Contributor
  • ***
  • Posts: 3523
  • Country: hr
Re: Keysight MSOX3104T Start-Up Problem!
« Reply #10 on: October 15, 2018, 06:08:42 pm »
As an embedded engineer, if it were me designing a piece of equipment like this I would have implemented a fail-safe way to update the FW even if the FLASH gets corrupted and the unit won't boot.  :-+

They did, in later version of firmware...
 

Offline GlowingGhoul

  • Regular Contributor
  • *
  • Posts: 236
Re: Keysight MSOX3104T Start-Up Problem!
« Reply #11 on: October 15, 2018, 06:09:31 pm »
I wasn't able to speak with a tech. They took my information and said I should hear from someone within 24hours. They had no idea what I was referring to regarding NAND corruption issues.

My warranty ended 2 months ago, and I don't have a good feeling about this.
 

Online HighVoltage

  • Super Contributor
  • ***
  • Posts: 4602
  • Country: de
Re: Keysight MSOX3104T Start-Up Problem!
« Reply #12 on: October 15, 2018, 08:00:32 pm »
My 3000X scope was almost 2 years out of warranty and Keysight fixed it for free, after I had send them the scope.
There are 3 kinds of people in this world, those who can count and those who can not.
 

Offline Sal Ammoniac

  • Super Contributor
  • ***
  • Posts: 1249
  • Country: us
Re: Keysight MSOX3104T Start-Up Problem!
« Reply #13 on: October 15, 2018, 08:11:46 pm »
As an embedded engineer, if it were me designing a piece of equipment like this I would have implemented a fail-safe way to update the FW even if the FLASH gets corrupted and the unit won't boot.  :-+

They did, in later version of firmware...

I'm talking about a hardware fail-safe here. A FW fix obviously doesn't help if there's no way to update the FW!
Complexity is the number-one enemy of high-quality code.
 

Offline 2N3055

  • Super Contributor
  • ***
  • Posts: 3523
  • Country: hr
Re: Keysight MSOX3104T Start-Up Problem!
« Reply #14 on: October 15, 2018, 08:58:37 pm »
As an embedded engineer, if it were me designing a piece of equipment like this I would have implemented a fail-safe way to update the FW even if the FLASH gets corrupted and the unit won't boot.  :-+

They did, in later version of firmware...

I'm talking about a hardware fail-safe here. A FW fix obviously doesn't help if there's no way to update the FW!

Well I guess they know how... I guess they setup secondary image or ECC or something. It can detect firmware corruption and starts backup boot procedure...
 

Online TheSteve

  • Supporter
  • ****
  • Posts: 3272
  • Country: ca
  • Living the Dream
Re: Keysight MSOX3104T Start-Up Problem!
« Reply #15 on: October 15, 2018, 09:02:44 pm »
The bootloader is in NOR flash, it loads the primary image from the NAND flash, if that fails it goes to a second image - in newer firmware anyway. So most of the time a network boot can recover the scope, unless some of the truly unique info was lost - model/serial, cal data, licenses etc. The newer firmware also handles remapping bad blocks so that both images are correct/clean.
VE7FM
 
The following users thanked this post: 2N3055

Offline GlowingGhoul

  • Regular Contributor
  • *
  • Posts: 236
Re: Keysight MSOX3104T Start-Up Problem!
« Reply #16 on: October 15, 2018, 09:37:02 pm »
The bootloader is in NOR flash, it loads the primary image from the NAND flash, if that fails it goes to a second image - in newer firmware anyway. So most of the time a network boot can recover the scope, unless some of the truly unique info was lost - model/serial, cal data, licenses etc. The newer firmware also handles remapping bad blocks so that both images are correct/clean.

I'm a bit confused. Are you saying that you need to network boot vs normal boot to recover, or that recovery happens automatically, in which case what is the relevance of network boot? I have ethernet on this scope.
 

Online TheSteve

  • Supporter
  • ****
  • Posts: 3272
  • Country: ca
  • Living the Dream
Re: Keysight MSOX3104T Start-Up Problem!
« Reply #17 on: October 15, 2018, 09:55:00 pm »
The bootloader is in NOR flash, it loads the primary image from the NAND flash, if that fails it goes to a second image - in newer firmware anyway. So most of the time a network boot can recover the scope, unless some of the truly unique info was lost - model/serial, cal data, licenses etc. The newer firmware also handles remapping bad blocks so that both images are correct/clean.

I'm a bit confused. Are you saying that you need to network boot vs normal boot to recover, or that recovery happens automatically, in which case what is the relevance of network boot? I have ethernet on this scope.

You'd need to open the scope and connect to the serial debug header. From there you can interact with the bootloader and tell it to tftp a prepared image via a network connection.
VE7FM
 

Offline Sal Ammoniac

  • Super Contributor
  • ***
  • Posts: 1249
  • Country: us
Re: Keysight MSOX3104T Start-Up Problem!
« Reply #18 on: October 16, 2018, 04:57:37 pm »
Spent 45 minutes last night repeatedly pushing the power button. No joy. I'll try the same tonight and if still no joy will pack it up and ship back to Keysight tomorrow morning. They quoted a 12-14 day turn-around time on the repair.
Complexity is the number-one enemy of high-quality code.
 

Offline GlowingGhoul

  • Regular Contributor
  • *
  • Posts: 236
Re: Keysight MSOX3104T Start-Up Problem!
« Reply #19 on: October 16, 2018, 06:26:43 pm »
It's been over 24 hours since I called Keysight and was told I'd be called back within that time. Not good.
 

Offline GlowingGhoul

  • Regular Contributor
  • *
  • Posts: 236
Re: Keysight MSOX3104T Start-Up Problem!
« Reply #20 on: October 16, 2018, 08:55:09 pm »
Just a quick update on Keysight's customer service. I just called, since they never called me back.

Bottom line, they will not assist, period. I'm on my own, 60 days out of warranty.

They denied knowing anything about 'corrupt nand', said there is no service note, and there is no 'known defect' of that type with any model.

When I asked about paid repair options, they demanded to see my origianl invoice, depsite purchasing the scope from Newark, and registering it with them the day it arrived, and purchasing numerous options that Keysight had to process.

Never again. A $5000 paperweight they refuse to repair. For what it's worth, don't tell them you are an individual, you MUST be a company if you want decent customer service. The moment I told them I had no company name, this was my personal scope, their attitude immediately changed from eager to help to suspicion. What the hell happened to Keysight?

« Last Edit: October 16, 2018, 08:57:45 pm by GlowingGhoul »
 

Offline JonM

  • Supporter
  • ****
  • Posts: 157
  • Country: us
Re: Keysight MSOX3104T Start-Up Problem!
« Reply #21 on: October 16, 2018, 09:13:52 pm »
Just a quick update on Keysight's customer service. I just called, since they never called me back.

Bottom line, they will not assist, period. I'm on my own, 60 days out of warranty.

They denied knowing anything about 'corrupt nand', said there is no service note, and there is no 'known defect' of that type with any model.

When I asked about paid repair options, they demanded to see my origianl invoice, depsite purchasing the scope from Newark, and registering it with them the day it arrived, and purchasing numerous options that Keysight had to process.

Never again. A $5000 paperweight they refuse to repair. For what it's worth, don't tell them you are an individual, you MUST be a company if you want decent customer service. The moment I told them I had no company name, this was my personal scope, their attitude immediately changed from eager to help to suspicion. What the hell happened to Keysight?

I am fairly certain that Keysight will fix this, but it might not get moving until Daniel Bogdanoff sees this thread. There have been multiple posts about the NAND corruption and it seems that Keysight fixed them all, even if past warranty period.

I just powered up my personally owned DSOX3024T to be sure that it is OK after sitting for a while, but I have recent firmware so it should not be an issue. I do wonder if I should get a service contract when my warranty runs out in another year. However, it's probably expensive for a lightly used, personally owned, non-profit instrument.

 

Offline GlowingGhoul

  • Regular Contributor
  • *
  • Posts: 236
Re: Keysight MSOX3104T Start-Up Problem!
« Reply #22 on: October 16, 2018, 09:26:30 pm »
I called back, livid. Keysight agreed to evaluate the unit for repair for a fee of $311, but I won't have to pay that until they give me the full quote for the repair (how generous), somewhere around $2000.

I was told there is no service note, no known widespread problem, no NAND fault repairs out of warranty.


« Last Edit: October 16, 2018, 09:31:30 pm by GlowingGhoul »
 

Offline sibeen

  • Frequent Contributor
  • **
  • Posts: 271
  • Country: au
Re: Keysight MSOX3104T Start-Up Problem!
« Reply #23 on: October 16, 2018, 09:33:50 pm »
GlowingGhoul, perhaps you should mention where you are based. Someone, probably Daniel, from Keysight will eventually see this or be pointed to it. They'd need to know where you are from so the arse kicking can commence.
 

Offline GlowingGhoul

  • Regular Contributor
  • *
  • Posts: 236
Re: Keysight MSOX3104T Start-Up Problem!
« Reply #24 on: October 16, 2018, 09:35:19 pm »
GlowingGhoul, perhaps you should mention where you are based. Someone, probably Daniel, from Keysight will eventually see this or be pointed to it. They'd need to know where you are from so the arse kicking can commence.

I'm in the US, and was dealing with customer 'service' at Keysight HQ.
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf