Agilent 54622A scope - 25% signal attenuation on CH1

[giosif]

Hi,

It seems my 54622A has developed a problem on CH1: some minutes after turning the scope on, the input signal gets attenuated by 25%.
So, the channel starts with displaying correct values right after powering on the scope, but it eventually begins attenuating the signal and then remains that way until I turn the unit off.
The colder the scope when turned on, the longer the period the channel works fine, before starting attenuating (so, suspect something influenced by temperature).

Has anyone encountered something similar and/or any idea where to start looking?

More observations on this:

• it only happens on CH1; CH2 is working fine
• the atenuation is proportional with the input voltage (DCV)
  
  • Actual - Displayed
  • 100mV - 75mV
  • 200mV - 150mV
  • 400mV - 300mV
  • 800mV - 600mV
  • 1V - 750mV
  • 2V - 1.5V
  • 4V - 3V
  • 8V - 6V
  • 20V - 15V
• the behaviour is the same with an inverted input voltage (i.e. reverse polarity)
• same behaviour with AC sine wave input signal and AC coupling
  
  • Actual - Displayed
  • 4V pk-t-pk - 3V pk-t-pk
  • 8V pk-t-pk - 6V pk-t-pk
• input impedance for CH1 is correct all the time - 1Mohm

Thanks!

[DaJMasta]

Sounds like the frontend amplifier(s) has got an issue with heat or a cold solder joint.  Characterizing the failure some more may help narrow things down:
Verify it's not caused by UI input, check if it starts reading low with just uptime and a signal applied
Verify it doesn't vary with frequency - push up towards the ceiling of the bandwidth and verify that it's still 25% down from the other channel
Verify the behavior when it switches to the low value - does it fall off slowly, does it intermittently read normal or high, is it just a sudden drop to 25% low
And check to see if anything else is effected - if you can't determine anything when it's all together normally, it could be worth monitoring voltages and trying to find a change when the issue happens - could still be power supply related, but a small rise could mean that an amplifier is being disconnected or something similar


Sounds to me like after the input protection and range switching, something in the signal conditioning path before the ADC is acting up.  It could ADC reference related, but I'd expect that to effect both channels.  Could also be that there's damage of some sort to a passive in the filtering in that section that shorts to ground making a resistive divider after it warms up.  If there's a programmable gain amp or attenuator after that initial range switching attenuation, it could be the control signals going to those parts either (if not the part itself).  There's a chance it's a digital thing too, I suppose, maybe a saved calibration going bad or something, but it doesn't sound like a stuck bit or something, so unless other people are reporting software issues, I wouldn't really expect it.


Certainly an interesting failure, but it sounds like it's a specific enough one that you can probably pinpoint the culprit.  A service manual is available, so running through some of the diagnostics in there could shed some additional light on the situation.

[JFJ]

... any idea where to start looking?

The power-on self test doesn't stand much of a chance of detecting the fault, because of the delay before the problem occurs. So, running the internal self tests when the fault is present, might provide a clue to its cause:

Press the Utility button, the Service softkey and then the Self Test softkey.

[giosif]

Thank you both for the pointers!
Let me go do these checks and will come back with the observations.

[giosif]

The power-on self test doesn't stand much of a chance of detecting the fault, because of the delay before the problem occurs. So, running the internal self tests when the fault is present, might provide a clue to its cause:

Press the Utility button, the Service softkey and then the Self Test softkey.

Just done that while the issue was present and self test was PASSED.

[giosif]

Ok, so I've tried a few things and, if anything, there are now more issues than before. 

• I confirmed the input signal stayed constant while the scope was reporting the signal going low (so, it is really the scope having the fault).
• I checked with a 20MHz signal (don't have anything than can output signal at a higher frequency) and attenuation rate remained the same.
• I confirmed that, when the attenuation starts, the signal falls slowly (no sudden drop) and the reading is consistent (i.e. not intermittently reading normal, then low, etc.)
• I checked all voltages mentioned in the service manual and all of them are within specs, with no ripple.
• I did a visual inspection of the input section under some level of magnification and it all looked normal (no suspiciously looking joints).
• With the case of the scope off and having a fan blowing air onto the IC's (as per the service manual), I started tapping the board and IC's in most of the areas (including the input section, but hitting the heatsink, rather than the components underneath). There was absolutely no change, not even a blip.
• With the above in mind, I started wondering whether this may be a calibration issue, rather than a "physical" problem, so I initiated a user calibration.
• On the first run of the user calibration, I got an error saying something like "channel 1". That was the first and only time I saw that error.
• Subsequent runs of the user calibration all failed, but with varying error messages: "gain" (most of the times) and "display linearity" (occasionally).
• Finally, I swapped all op-amps in the input sections between CH1 and CH2 in the hope that the user calibration would then fail for CH2, but that didn't make any difference.

Last thing I'm left wondering is whether the relays in the input sections might have worn out, but finding replacements for them to check this theory is not going to be easy or cheap...
Not sure that is worth the time and money...

Sounds like the frontend amplifier(s) has got an issue with heat or a cold solder joint.  Characterizing the failure some more may help narrow things down:
Verify it's not caused by UI input, check if it starts reading low with just uptime and a signal applied
Verify it doesn't vary with frequency - push up towards the ceiling of the bandwidth and verify that it's still 25% down from the other channel
Verify the behavior when it switches to the low value - does it fall off slowly, does it intermittently read normal or high, is it just a sudden drop to 25% low
And check to see if anything else is effected - if you can't determine anything when it's all together normally, it could be worth monitoring voltages and trying to find a change when the issue happens - could still be power supply related, but a small rise could mean that an amplifier is being disconnected or something similar


Sounds to me like after the input protection and range switching, something in the signal conditioning path before the ADC is acting up.  It could ADC reference related, but I'd expect that to effect both channels.  Could also be that there's damage of some sort to a passive in the filtering in that section that shorts to ground making a resistive divider after it warms up.  If there's a programmable gain amp or attenuator after that initial range switching attenuation, it could be the control signals going to those parts either (if not the part itself).  There's a chance it's a digital thing too, I suppose, maybe a saved calibration going bad or something, but it doesn't sound like a stuck bit or something, so unless other people are reporting software issues, I wouldn't really expect it.


Certainly an interesting failure, but it sounds like it's a specific enough one that you can probably pinpoint the culprit.  A service manual is available, so running through some of the diagnostics in there could shed some additional light on the situation.

[DaJMasta]

A gradual drop off with time basically rules out a digital failure, and gain related self test failures are exactly the kind of thing you'd expect to see if an amp wasn't working.  Not reacting to temperature changes suggest that it's not just a part heating up and stopping normal function, but especially if under some parts it's not entirely ruled out.  The big tell here would be that if you could get the failure to occur, then you could aggressively cool the frontend and get it to at least start moving in the opposite direction, then you'd be able to confirm if it's thermal related.

Swapping amps would rule them out in theory, but only if you swap all of them between the input switching and the ADC, since even the buffer amp for the ADC could do this, and some of them may not be opamps.  Checking voltages is still a good choice, if you can probe the local regulation for the frontend and the ADC's voltage reference that will be more useful and they may not match the board's primary rails.


Then there's actually a physical method that could work..... leave the top/can off and start gently poking parts in the signal path for channel 1 with a non-conductive probe.  If you can get a reaction, there's a chance a small crack or defect in that part is making it fail partially over time.


Definitely a more time consuming thing to chase down, so maybe not worth the time, but this kind of thing also tends to be rewarding if you can get it fixed.  While a lot of parts should still be available off the shelf, the source for any of the custom/hard to find stuff would probably be a donor unit rather than trying to find new or new old stock replacements.

[MarkL]

Since you have one working channel (CH2), you might want to try applying the same signal simultaneously to both CH1 and and CH2 input BNCs.  Then start comparing the same points on each channel to eachother, starting from the ADC inputs (U1600 and U1800) and work your way to the BNCs.  The layout for the two channels look identical so this shouldn't be too difficult even without a schematic.  You should be able to spot where the 25% discrepency creeps in.

Although you might find the issue, I wouldn't start randomly swapping components between the two channels unless you've exhausted all other options.

[giosif]

That is something I had considered at some point, but the fact the service manual mentions having to use a fan just for taking the scope case off, I was afraid of running the unit also with the heatsink covering the first part of the inputs removed.
And, if I don't remove the heatsink, I can't access most of the inputs area.

But, I guess, the alternative is having the scope in this "half-dead" state, so probably worth the risk of running the scope without that heatsink and checking as per your suggestion.

Since you have one working channel (CH2), you might want to try applying the same signal simultaneously to both CH1 and and CH2 input BNCs.  Then start comparing the same points on each channel to eachother, starting from the ADC inputs (U1600 and U1800) and work your way to the BNCs.  The layout for the two channels look identical so this shouldn't be too difficult even without a schematic.  You should be able to spot where the 25% discrepency creeps in.

Although you might find the issue, I wouldn't start randomly swapping components between the two channels unless you've exhausted all other options.

[giosif]

Cooling off parts of the board is something that did cross my mind.
However, I first need to get a freeze spray (any recommendations, BTW) and, second, I need to understand one thing: when freezing electronics, what are the chances of creating a short circuit?
I know that, as a general rule, ice is not a good conductor, but how about when the ice just about starts to melt?
Could it not start melting right near the circuit components, before it becomes visible on the surface as well (i.e. when I can see it as well)?
Or is it that even the melted ice is not conductive?

On voltages, I agree, but the only one not mentioned in the manual but which is on the board is a -5V rail.
I'll have a closer look again, maybe I missed other rails.

For the physical check, as I mentioned to MarkL, I was afraid of running the scope without the can on the two IC's it is intended to provide cooling for.
I probably need a bit more courage with these things... 

Thanks for the encouragement on trying to fix this!
My enthusiasm with this repair was running low, but I'll keep at it, even with the risk of burning those IC's I mention above.
A donor unit would be really good, but the prices on these scopes, even non-working, are silly.

A gradual drop off with time basically rules out a digital failure, and gain related self test failures are exactly the kind of thing you'd expect to see if an amp wasn't working.  Not reacting to temperature changes suggest that it's not just a part heating up and stopping normal function, but especially if under some parts it's not entirely ruled out.  The big tell here would be that if you could get the failure to occur, then you could aggressively cool the frontend and get it to at least start moving in the opposite direction, then you'd be able to confirm if it's thermal related.

Swapping amps would rule them out in theory, but only if you swap all of them between the input switching and the ADC, since even the buffer amp for the ADC could do this, and some of them may not be opamps.  Checking voltages is still a good choice, if you can probe the local regulation for the frontend and the ADC's voltage reference that will be more useful and they may not match the board's primary rails.


Then there's actually a physical method that could work..... leave the top/can off and start gently poking parts in the signal path for channel 1 with a non-conductive probe.  If you can get a reaction, there's a chance a small crack or defect in that part is making it fail partially over time.


Definitely a more time consuming thing to chase down, so maybe not worth the time, but this kind of thing also tends to be rewarding if you can get it fixed.  While a lot of parts should still be available off the shelf, the source for any of the custom/hard to find stuff would probably be a donor unit rather than trying to find new or new old stock replacements.

[DaJMasta]

You can try to just use a bigger fan or a higher setting.  Not heatsinks is never a great option, but sometimes it can be good enough.  If you have heatsinks lying around, it could be that even a smallish one installed temporarily will allow you to run comfortably under the one fan.

I've had the larger format LeCroy fans open with a box fan blowing on them - so much more airflow than the original that it worked great, but that was with heatsinks attached and I was able to monitor temps with a thermal camera to verify before I left it running long enough to work on.

The sort of latent aspect to the failure does make it trickier, though, since you can't just power up, measure, then turn off before heat becomes an issue.



Though, I guess there's an alternative to that: mod wires.  Some little wires soldered to the points you'd want to probe, a bit of tape to keep them from moving around excessively, and now you have probing points you could even run outside of the chassis with it running inside.  Not as useful for dynamic signals (lots of inductance in that loop), but good for voltage monitoring.


It can certainly be discouraging, I've had more than my fair share of projects stall out (and remain shelved...), but until you trace a fault to some unobtanium part or some wire-bond-assembled-in-a-clean-room sort of assembly, the odds that a fix is possible is still reasonable, even if it takes forever.  An advantage of doing things for a hobby is you can indulge in that, and if you end up with a working scope, you can take pride in the repair and probably know a lot about how it works.

[No.Mad]

Hey giosif,

Regarding cooling spray, grab that one: https://uk.rs-online.com/web/p/freezer-sprays/3292707/
We use it in work for thermal testing in R&D. As you are UK based RS should be know to you

Few pointers: take off original heatsink, apply fresh thermal compound and fit these bad boys: http://3dmodularsystems.com/en/electronic/140-heatsink-for-a4988-or-drv8825-stepper-motor-drivers.html

Then blow air directly with a fan on board during power up and measure CH1 and CH2 voltages. It would be nice to hook up thermistor or thermometer to control how hot it gets and keep Freeze Spray next to you in case of emergency. It can work a bit without case and heatsink, be brave!

My guess is: due to age and/or bad thermal compound on heatsink pillars heat from Lucent IC's is not properly transmitted. That can affect this IC directly or, due to increased thermal stress, any other component nearby because of thermal drift. But, it's only hypothesis. Would be nice to see pics of these two analogue channels - if you could.

Regards

[giosif]

You can try to just use a bigger fan or a higher setting.  Not heatsinks is never a great option, but sometimes it can be good enough.  If you have heatsinks lying around, it could be that even a smallish one installed temporarily will allow you to run comfortably under the one fan.

I've had the larger format LeCroy fans open with a box fan blowing on them - so much more airflow than the original that it worked great, but that was with heatsinks attached and I was able to monitor temps with a thermal camera to verify before I left it running long enough to work on.

The sort of latent aspect to the failure does make it trickier, though, since you can't just power up, measure, then turn off before heat becomes an issue.



Though, I guess there's an alternative to that: mod wires.  Some little wires soldered to the points you'd want to probe, a bit of tape to keep them from moving around excessively, and now you have probing points you could even run outside of the chassis with it running inside.  Not as useful for dynamic signals (lots of inductance in that loop), but good for voltage monitoring.


It can certainly be discouraging, I've had more than my fair share of projects stall out (and remain shelved...), but until you trace a fault to some unobtanium part or some wire-bond-assembled-in-a-clean-room sort of assembly, the odds that a fix is possible is still reasonable, even if it takes forever.  An advantage of doing things for a hobby is you can indulge in that, and if you end up with a working scope, you can take pride in the repair and probably know a lot about how it works.

Sorry, replying after a long pause!

Yes, have thought of mod wires but, without knowing the specific area where to probe, I would end up soldering lots of wires all over the place and will most likely lose track of where each of them connects.

Thank you for the encouragement!
And you are right: I'm not giving up on this yet.
Also, for some reason, I quite like this model of scope.
So, I'm keen to get this back to working state.

[giosif]

Hey giosif,

Regarding cooling spray, grab that one: https://uk.rs-online.com/web/p/freezer-sprays/3292707/
We use it in work for thermal testing in R&D. As you are UK based RS should be know to you

Few pointers: take off original heatsink, apply fresh thermal compound and fit these bad boys: http://3dmodularsystems.com/en/electronic/140-heatsink-for-a4988-or-drv8825-stepper-motor-drivers.html

Then blow air directly with a fan on board during power up and measure CH1 and CH2 voltages. It would be nice to hook up thermistor or thermometer to control how hot it gets and keep Freeze Spray next to you in case of emergency. It can work a bit without case and heatsink, be brave!

My guess is: due to age and/or bad thermal compound on heatsink pillars heat from Lucent IC's is not properly transmitted. That can affect this IC directly or, due to increased thermal stress, any other component nearby because of thermal drift. But, it's only hypothesis. Would be nice to see pics of these two analogue channels - if you could.

Regards

Hi,

Thanks for the tip on the cooling spray, but it shows up as "discontinued".
In any case, I bought another similar spray, so I'm good on that now.

Back to investigating the problem, I think I need to revise my initial description of it: I am not that sure anymore there is a correlation between the problem and temperature.
I am saying this because, while it is true that, on initial power up of the scope, all works fine (including calibration) and then, after a while, I am starting to get the attenuation on CH1 (and calibration fails), the interval it takes from working to experiencing the issue varies.
What I mean is: if I leave the scope powered off for a long time (e.g. weeks), the first time I power it on, it will take a longer time until the issue starts showing up (e.g. 10-15 mins or so).
If I then power off the scope and leave it off for a day, when I power it back on, the issue will start showing up sooner (e.g. 5 mins or so).
I can't imagine something in this scope taking longer than a few hours to cool down, to account for the above difference.

In fact, I think I might be looking at the wrong side of the problem altogether: I am tempted to think the attenuation I am seeing on CH1 is likely to do with the last successful run of the calibration.
And since I can perform a successful calibration only when the meter has just started (after sitting powered off for some time), the components haven't had the time to warm up properly.
As such, the calibration is relevant only for when the scope hasn't warmed up.
Now, there is a snag to this theory, mainly that CH2 should also behave in similar manner, since the calibration is for both channels, but maybe there is some explanation to that (which I can't think of, at this stage).
So, maybe I need to focus my troubleshooting on the calibration failing once the scope has warmed up.
And the failure message I get when that happens is "Gain" as the reason.

Anyone with any idea where to look for such an error?

Thanks!