Keysight MSOS104A Repair Attempt

[Extrasolar]

I found a MSOS104A 1GHz Oscilloscope from Keysight. It was marked as offline and not working so I thought I could try to repair it and eventually picked it up and set it on my bench.
The problem seems to be related to the timebase clock and PLL2 as we can see on the image.
"Timebase error detected in PLL2", perhaps a missing clock, maybe some buffer decided to take a vacation, or the internal 10MHz Oven Controlled Oscillator is bad...
By feeding an external 10MHz Reference and the specified magnitude and changing the trigger type on the horizontal scale, for a brief moment I was able to get a Square Wave on the screen as Ch1 was connected to the Cal terminal on the front panel. So, maybe there is a bad buffer or routing mechanism for the clock (I'm just assuming at this point) that is intermittent...
There is absolutely nothing online on the public domain except for a "General Service Guide" which is really reduced to only "Board Swap" kind of repairs.
The Acquisition Board looks really intimidating but hopefully the problem might be isolated only to the Timebase section.

[Extrasolar]

So, There are 4 Clock related sections, a clock Ref (10MHz) output, Oven Controlled Oscillator , Main VCO and PLL gen and a negative 11.5V Oscillator made of discrete components.
I've tested the OCO and by powering it up locally and I was able to see an output at 10MHz which confirms this section to be working (also I checked a few buffers and all working).
For the moment will continue to the VCO section.

[Extrasolar]

I tried to do the same with the VCO, 5V rail (was pulling 5mA), Applied a Voltage on the Tuner (PIN2) and the SMU went in Current compliance (it was pulling 100mA at 2.1V) 
To me this is a good indicator that the VCO is bad. I decided to remove it and try to power up in a different environment.

[KE5FX]

 I can't see what the PLL chip part number is, but it's possible that the VCO tuning line simply went straight back to it with the loop filter components in parallel.  PFDs with charge pump outputs often don't use series R, and in that case, you might have forward-biased some protection diodes somewhere. 

[Extrasolar]

I desoldered the lid or the shield of the VCO, but managed to knock out a few components, no problem tho because I got everything back and soldered.
My suspicion that the VCO was toasted got confirmed, the part was pulling nothing, and I was not able to get an output from it.
Here is also the Diagram of the Crystek CVCO55 602934 1702-02 (seems a custom part number as I was not able to find this part online).
I have ordered a few Crystek VCO with output ranging from 500MHz to 2GHz, with the same form factor, lets hope for the best at this point 

[Extrasolar]

The VCO is driven by a LTC6945 Ultralow Noise and Spurious 0.35GHz to 6GHz Integer-N Synthesizer, Neat !! (I have a few of those parts !!)
I was able to RE the schematics of the synthesizer block.
Unfortunately I must wait for the VCOs to continue this repair, but in the meantime I will continue to probe around the Timebase Section, just to rule out any other problem it might have ...
Any help on finding the datasheet of the VCO will be greatly appreciated !!

[gamalot]

The component with 'UA' top marking on it looks like a RF BJT 2SC5606, not a NMOS.

https://vakits.com/sites/default/files/2SC5606.pdf

[Extrasolar]

True about the Transistor, thank you Gamalot !!
I tried today to fix the VCO and kind of succeeded, well sort of !!
The VCO outputs around 840MHz, when powered up correctly and by correctly I mean had to short the capacitor at the base with tweezers !!
It works, but has a lot of phase noise, unstable and intermittent operation.

[picburner]

What you have indicated as C5 most likely is an inductor or resistor instead.
The base/gate of 'UA' could not be polarized otherwise, that's why shorting it with the tweezers the circuit starts to work.

[wraper]

What you have indicated as C5 most likely is an inductor or resistor instead.
The base/gate of 'UA' could not be polarized otherwise, that's why shorting it with the tweezers the circuit starts to work.

IMHO he probably switched some components while resoldering them, thus ended up with capacitor on the base (if it's 2SC5606) as by its looks it's certainly a capacitor.

[Extrasolar]

Hi wraper,

There were 3 parts that were knocked off when I removed the lid.
I opened up a similar VCO and saw the Resistor (820 Ohms) forming a divider with a 1.5k resistor. The inductor is a larger package than the capacitor and there was only one pad that could actually accommodates it.
I think there might have been a resistor as well in parallel(on top) of the capacitor at the base.
Nonetheless, will try to rearrange the inductor to see what happens ...

[gamalot]

Some SMD multilayer inductors look similar to MLCCs.

[Extrasolar]

The VCO is now working. I soldered the inductor to the BJTs base. Its operational from 4.635GHz with 0.5V to 4.703GHz with 4.7V at the tuner.

[emoco]

Hi,
I just got the same warning on my scope. Did the VCO solve the problem?

[HighVoltage]

Nice repair!

Any idea what the failure mode was for the VCO?

[Extrasolar]

Hello,

Sadly the VCO did not resolve the problem, but on the bright side, I now have it characterized completely 
I've swapped the LTC6945 Synthesizer with a new one and still did not work.
I am going to investigate further at the output of the Synthesizer, because somehow I have a feeling that some buffer might have failed   
It has to be for sure a HF switch or buffer at some point, because by feeding an external 10MHz signal, I was able to briefly see the Cal Out quadrature appear on the screen.

Regards

[emoco]

Thank you for the update.

[emoco]

If it is of any help. After having my DSOS104A disassembled, connectors removed, and assembled again the error ""Timebase error detected in PLL2" did not appear. But the message comes back occasionally when the scope is started.

[Extrasolar]

Intermittent problems, hmmm this is a bit problematic, it is a strong indicator that probably a power rail on the PLL2 is in fact intermittent !!
I will check a few signals on the Timebase as I can see the PLL2_Lock LED indicator does not turn on. Again I suspect that some signal is not present.
There is a Fully Differential amplifier that is polarized only by a negative 3.3 rail (very strange  ) that "Buffers/Amplifies" the Synthesizers output (4.64GHz to 6GHz). I have prepared a HF probe to check for the input - output and will post updates soon.

[BeamDump]

Lots of interesting discoveries here. Thank you for putting in the effort.

In the processes of repairing one my self. Same error.

From power ON only the PLL1_lock LED is on but once the software starts PLL2 comes on too.
Measured RF output from the VCO, during OS boot it sits at 4,950GHz, once the soft starts loading and PLL2 LED comes on, frequency jumps to 4,978.900 and oscillates between that and 4,979.800. Not sure how there can be a lock in this case.
5.12V on VCC, 4.6V on Tuner

Any idea where it's possible to get a similar VCO? It's Keysight authorized part and Crystek won't sell without their permission. Have some contact in there, see how that goes.

[Extrasolar]

Hi there !!

I've made some progress on this and reverse engineered the entire signal chain from the OXO to the output of the Synthesizer.
I will be probing around now as I strongly suspect that either the fully Differential Amplifier next to the Synthesizer or the OXO output buffer (ref input of the synthesizer)is bad.
In the case of the first problem, I am putting up together a simple passive RF probe and will be using a Spectrum Analyzer to "see" the signal.
The OXO buffer is easy enough to be probed with a normal scope. One thing that is absolutely needed is a cooling fan on the ADC, otherwise it will get toasted.
Will post the results this week !!

Regards

[BeamDump]

I would say don't run the acquisition board without the fans hitting it's full surface, apart from the ADC every other heat sink and memory get's roasty toasty fairly quickly.

In my case it's definitely the VCO, output is sporadic. Measured in the acquisition board and out. Too bad i don't have a signal or spectrum analyzer that can go over 4 GHz but my 20GHz CSA8000 does the trick to "see" the signal. Interestingly that with such a mess the board still lights up the PLL2_Lock LED.



Ordered a few Crystek VCO's in this range and some LTC6945's for experimentation. Otherwise conversation with Keysight's Finish department has reached a dead end. Part can't be sold and can't be authorized for selling by the Crystek. Instead send the scope in for "repair". Further clarification on "repair" - they don't do any component level repair and acquisition board gets swapped. 

[vtp]

One of the Infiniium S Series scopes in office threw that error some months ago.

It was repaired under factory warranty and apparently is a known problem to KS.

Excerpts from service note:

The Problem - Under certain circumstances a PLL2 error may occur on the S-series products.

ADMINISTRATIVE INFORMATION
------------------------------------
Action [X] On specified failure

Location category: [X] Service center

Availability: Product's support life
No charge available until: December 31, 2022

Just FYI.

[BeamDump]

Probably. A colleague with an S-series had a PLL2 error today. Guess it is plagued with this issue. Wonder if it's Keysight or Crystek to blame. If they are well familiar with this, then why indeed sell a ~50EUR part when they can charge 6K for repairs.

If I'm to believe what I've been told, this is suppose to be 5GHz VCO. 

Desoldered the transistor from my VCO:

[Extrasolar]

Hi,

I can fully characterize the VCO, frequency bandwidth and Voltage control range.
The only problem is that my VCO might be faulty as well 
Last time I checked it was spot on 4.6... GHz and remained stable with no noticeable phase noise, 0.5V to 4.7V Control. (removed from Acquisition Board).
I have attached the Synthesizer RE Schematics below, the circuit if fairly easy to follow... My question is, where is the PLL2 synthesized/generated ??
There is another Synthesizer on the board, slightly different part but overall same functionality (which I haven't checked yet... )
 

[BeamDump]

Hi

What do you mean? It's generated in LTC6945, it's what it's suppose to do. Pin3 (status), which in this case has to be programmed through SPI as lock, even goes through resistor to the base of MMBT3904LT1 that goes to PLL2_Lock LED. 

4.6GHz is low for this VCO, it's suppose to be 5.

[Extrasolar]

Hi

I was able to trace the status Pad of the synthesizer to PLL1, and was unable to find the path for PLL2 !!

[BeamDump]

Hi

I also can't find any trace from the 2nd output of the buffer. Then the lock status is communicated through SPI?

Another question, when the PLL2 error pops up, are you able to perform any software function? Self test or anything at all.

[Extrasolar]

Hi

No it does not do anything in the sense that after the autocal routine it sprays out the same error and does not do anything. You can try an external Reference 10MHz and change the horizontal trigger.
I got it working for an instant by doing that. Another thing is that I could not detect the 10MHz reference at all on the board. Now all my focus is on the signal path from the oven controlled oscillator to the REF input of the Synthesizer.

Kind regards

[BeamDump]

Hi

Replaced the transistor in my VCO, it is a 2SC5606. Now the output is clean, stable and sits at 5.0GHZ at 4.3V. Plugged it back in to the board, PLL2 is locked and the LED is considerably brighter than it was before but the software error remains.
Since i don't see a test point for Y2 buffer out for PLL2 status and probing front and back was unsuccessful i can only assume it's sending the status via SPI. Then why bother routing pin 3 of the synthesizer to the buffer if the output wont be used...

Ordered spare LTC6945's and evaluation/communication boards to better understand the synthesizer. Buffers are difficult to find. Ordered some from the one place i could find them in China, see how that goes.

I'll try your method.

[Extrasolar]

Hi,

Thats very interesting !! Thank you for the information. Does any trace appear on the screen ?? 
If yes, then I believe you can factory restore the scope at this point, to clear the error message !!

[BeamDump]

Hi

Haven't tried yet. I've been waiting for parts to arrive and they finally have. Shipping times are horrible and companies stopped caring in the slightest.   
Got some spare LTC6945's and NC7WZ17 buffers. I'll post any updates later.

[max-bit]

Here, the control of the frequency synthesis system is probably damaged, and not the PLL (LT..) or VCO itself is damaged. Therefore, the system does not hold the frequency.
Check how the LT IC is controlled and whether there are control signals at the LT IC inputs.
That would explain why the oscilloscope ended up in the trash.
If the damage was simple, Keysight would fix it, but due to the FPGA (and / or ASIC) system, unfortunately, they do not replace a single IC, but replace the entire board.
And that probably costs as much as the entire oscilloscope.

[BeamDump]

Any FPGA/ASIC can be replaced. Maybe not in EU anymore, since most factories that would do such repair are either gone or won't provide the service. I replaced an A-ASIC in a Fluke 199c few years back. True, Fluke sold the IC for 120EUR but was possible.

All the frequency synthesis happens in the LTC6945 for the time base, that's it. It does have SPI where of course it can be programmed and read out.

They could easily sell the IC, knowing where the problem is but instead it's more profitable to them to charge 6K for initial inspection and possible board swap. Same reason why these scopes have no schematics or part lists.

Just got everything i need, now to get some time with it. December is the worst time.

[Extrasolar]

Hi All,

A second MSOS804 just failed at work, with the same problem !!
Will capture the SPI protocol as I have a few evaluation boards with the same synthesizer.
I could not find any problem on the signal chain from the oven to the synthesizer. Anyone knows the part number of the fully differential amplifier which is right after the Synthesizer (U33003) ??
I cannot find the Positive Supply for this Amplifier !!!!

[TheSteve]

Maybe temporarily add one to see what happens.

[KE5FX]

That might be an LMH5401 or something in the same family.  It may use both positive and negative supplies (e.g. -1.8 and +3.3).

[BeamDump]

Slightly better photo.

Sounds like a repeat of my story lately. Same thing with this one, no lock on PLL2?

[gamalot]

It looks like fanout buffer HMC720

https://www.analog.com/media/en/technical-documentation/data-sheets/hmc720.pdf

[BeamDump]

Yeah, all the pins match and it works only on a negative supply.

Extra, can you check the eye diagram coming out of yours?

[Extrasolar]

Hey,
Wow, mystery part no more , I have ordered a few HMC720s !!
I've received an ultra high bandwidth scope last week and will absolutely probe everything possible on that motherboard

So, on a systematic approach:
      1. Check Oven and signal path up to the synth.
      2. Check VCO
      3. Check Synth and Buffer
      4. if signal (+/-) is present at Buffers output (both outputs) then:
              Check  SPI comms.
      5. Check FPGA/ASIC 

Last week the same story repeated, one other MSOS804  broken with the same problem, and also a few weeks ago another MSOS404 same problem.
I wonder if Keysight uploads some artificial errors (functions) with updates on such scopes

[BeamDump]

Hi

Was thinking the same, that a flag went high somewhere during an intermittent event or maybe with an update. Can't speak about Keysight but one known offset printing quality control system was caught doing so.

So far swapped the n-synth and dual buffers, same thing. External ref also didn't show anything unfortunately.

What's the date on yours and what soft revision are they running?

[HighVoltage]

Last week the same story repeated, one other MSOS804  broken with the same problem, and also a few weeks ago another MSOS404 same problem.
I wonder if Keysight uploads some artificial errors (functions) with updates on such scopes

Too many problems with these S-Series scopes.
I always wanted one, maybe that is not such a good idea anymore.

Thanks for all the great info in this thread.

[Extrasolar]

Hi,
I dont recall the version and to get it I would have to reassemble the whole thing.
HMC720 chips on the way (the percs of being ADI employee ).

[BeamDump]

Hi,
I dont recall the version and to get it I would have to reassemble the whole thing.
HMC720 chips on the way (the percs of being ADI employee ).

Some perk there 
I ordered 2 my self. Price is really something on these. Then again, not your run of the mill part.

With that, would like to advice to stay away from the Chinese. Good thing i ordered two sets of NC7WZ17P6, one from their bay, other from sets of evaluation boards. Wondered why theirs didn't work. Just a case with pins sticking out and nothing inside. After i bought the 1st 5 pcs for 3EUR, price jumped to 42EUR.

[Extrasolar]

Hello,

I've replaced the HMC720 and now the PLL2 is intermittently locking. which means that the level of the signal is low despite being on the right frequency.
There is a dual buffer that is on the OCO output (50MHz) which seems to be bad, as the output drops a lot compared to the input.
I've discovered that the output from the VCO is bad (see photos).
I was probing with a high bandwidth oscilloscope about (6GHz), and I could verify the presence of the signal beyond the dual buffer at the output of the Synth.
So I am suspecting 2 things now, VCO improper output and maybe some sort of buffer for the PLL2 beyond the HMC720.
Will check it later this evening and then will also check the Synth SPI coms.

[seebeck]

The same old problems would surface, Keysight should pay attention to his product quality！

[Extrasolar]

Hi guys,

Hmm, I brought it back to life, well, for a short while !!
Most definitively, its a power issue. We have been looking in the wrong place so far.
Replace all the Power Rail Caps ( tantalum mostly ) around the timebase !! I have ordered a bunch and I'm impatiently waiting to perform the recap

Cheers !!!

[Extrasolar]

Second Unit in Operation after 1 Hour still no PLL2 Error !!

[HighVoltage]

Nice find.
Power Rail Caps, especially tantalum types are known to fail in test equipment. These should be visible in the ripple of the rails.

[BeamDump]

Good find.
Still going?

[seebeck]

Hi Extrasolar,

The same PLL2 issue happened to my Keysight DSO-S 404A Digital Storage Oscilloscope.
You mentioned a replace of all the Power Rail Caps（tantalum mostly） around the time-base area, was it just that?
Thanks a lot.
I will try to repair my DSO this weekend.

With best regards.
Seebeck.

[veloyage]

Hi all,
TLDR: I just made an account to tell you that Keysight seems to repair this issue for free
We have a DSOS054A which has been sitting unusable with this error for a year or so. I looked around and found only this thread, with repair seeming very difficult, so it sat some more. With the note about a recap being successful I recently opened it up, checked it out, and put some tantal caps into my mouser cart. Before actually doing it though I wanted to hear from Keysight what an official repair would cost, and while going to create a service request I found some service notes which detail the repair procedure (board swap), the most recent one (https://keysightsales.file.force.com/sfc/dist/version/download/?oid=00D1a000000awb5&ids=0685a00000nYQsW&d=%2Fa%2F5a000002gh2L%2FmeC7tIclK9L9B1_CDON938jLsjuTkG8X0XQbeUeKEKg&asPdf=true) saying "NO CHARGE AVAILABLE UNTIL: December 31, 2023".
So I wrote them and got told to send it to DHL for forwarding and subsequent repair "on warranty". The actual warranty ended years ago, but for this known production issue they seem to not care about that.
I am not sure if the "no charge" will actually expire or they will extend it again, as they seem to have done before.

Hope this helps someone,
Thomas

[HighVoltage]

 "NO CHARGE AVAILABLE UNTIL: December 31, 2023".

Hope this helps someone,
Thomas

What a nice surprise, good to know.
Keysight sometimes is very generous with scope problems, even if they are out of warranty.

[BeamDump]

Haven't  tried to continue  repairing mine yet. So much work,  it's nearly impossible to find free time for  anything else anymore.

Last time i brought this problem up with Keysight they quoted me an initial 6000EUR cost for replacing acquisition board. I've sent them  another request referencing the attached service note.  Let's see what happens.

[BeamDump]

Well i can safely say that replacing all tantalum caps on board, doesn't solve the PLL2 problem. Still same effect, lock light on but soft start with fault and nothing past that.

Thanks to Extrasolar but not the solution. 

[Extrasolar]

Hello All, It has been a while !!

I restarted my repair on the MSOS104A. Recaping does not solve the issue. It is a serious problem that perhaps has a lot more to do with Firmware !!
I was told that a recent Firmware update, effectively solved this issue on a known PLL2 Timebase failure unit.
I am going to try to reinstall windows and the scope drivers from the most recent firmware update from Keysight.
For some strange reasons, the SSD on the scope broke all of a sudden and there are only 2 options to save it now : 1: install from scratch and hope for the best, 2:send to Keysight and pay ...
I will update the progress soon ...

Kind Regards

[Extrasolar]

Hi All,

I successfully installed the instrument drivers on a freshly installed Win 10 system. The acquisition board is recognized properly and the software works fine. However the PLL2 problem persists...

I have managed to prove that the scope actually works when we operate it remotely via USB:
There are different waveforms I wanted to test and curiously enough the scope followed with the :DIGITIZE command.

This makes me think the problem is actually some sort of firmware/software bug.

Kind Regards

[BeamDump]

I got to talking with Keysight. PLL2 light will come on even if there is problem with VCO and the entire issue here is the VCO.

What they told me: " The VCO and perhaps the synthesizer that drives it. But right now VCO is the concern as it swings widely in the MHz on the output. Even though board shows it's in a lock, it's obviously false.