Series defect on agilent 167xx boards?

[DoricLoon]

Thank you very much for the help Mark.

With the information you passed on earlier I discovered that the 7k5 resistor in the divider for U29 was in fact measuring 8k. Swapped the resistor and I have fully functional 16752A

Beauty.

Came on to post my results and I see your further post which has the information I had just noted down about the DAC to comparator pin out.

So for my low order bits on pod 4. Comparator U29 (pin39) to junction of (560R - 7k5) - DAC U39 Voutg (pin34), trace goes pretty much full length of the card.

Now to clean up the area's where the runners were and stick them on with new adhesive. I purchased a roll of 3M VHB clear foam tape 6mm wide 1mm deep. Hopefully it will play nicely with the PCB?

Thanks again 

[MarkL]

Great - glad you found the problem!

That's what I was going to try too - the thin 3M VHB tape, but I have some of the gray variety sitting around.

For now, after cleaning the boards, I left the runners off and I'm being careful with them.  I'm probably going to regret it after I rake several components off the bottom, and THEN I'll put the runners back on.

[keitheevblog]

Thanks so much for this dialog --- it's nice to see such detailed posts and information!

I'm looking for an appropriate replacement adhesive for the runners.

How's that 3M stuff working out? Have you found any adhesive that claims it is specifically FR-4/PCB safe?

Keith

EDIT: Sorry I know the thread is about a month old.....not too too stale, I hope.

[DoricLoon]

I've still not got around to sticking the runners back on. Roll of tape at the ready but that's as far as I've got. When I read the data sheet it sounded like the 3m tape should be safe? I guess that any issues with it would take a long time to transpire anyway. I did wonder if it would be a good idea to put on conformal coating first, just incase the adhesive doesn't play nicely?

[keitheevblog]

Yeah conformal coating down first sounds like a good idea.

My question was whether the pro-active approach to runner removal is a good idea or not. In boards where there are no visible signs of corrosion that are otherwise self-testing fine?

It occurred to me while I was removing them that you could possibly damage a board that worked fine.

Do you guys have an opinion on that?

I've been using a heatgun setting on low and trying to evenly heat just the runner. While the heat helps the adhesive release, I worry about these super tiny 0201's (or whatever) accidentally re-flowing. I'm guessing the surface tension would keep it in place.....but who knows?

My technique has been to use a plastic pry tool to get under an edge, and then wand the heatgun back and forth over the runner, all the while applying a lifting force. The runner does sometimes "pop" violently off the board.

I have noticed that the adhesive tape is both still sticky and flexible.....not hard.... and that I've been able to, in 95% of the cases lift the adhesive tape AND the plastic runner off fully intact.

Am I doing this right?

Thanks

[MarkL]

...
My question was whether the pro-active approach to runner removal is a good idea or not. In boards where there are no visible signs of corrosion that are otherwise self-testing fine?

It occurred to me while I was removing them that you could possibly damage a board that worked fine.

Do you guys have an opinion on that?

In general, I'm strong a believer in "if ain't broke, don't fix it,", which includes things like re-capping for no demonstrated reason.  I've seen too many people damage perfectly working equipment.

However, in this case my opinion is to remove the runners proactively.  I have somewhere around 25 cards, and ALL of them had corrosion on them, some severe and some only in a few spots.  For several years I was in the the "not proactive" camp until two cards I had received working and had been using were suddenly dead one day.  I took the runners off and several traces had been eaten clean through.  Fortunately, repairing the traces brought them both back.  Not wanting to repeat this fate, I removed all the runners on all my cards and scrubbed the areas with isopropyl.

No additional failures yet, but this was fairly recent within the last year or so.

I was careful but not overly paranoid about scraping off SMD components.  I have at least two of each card type so I could measure the value on the other card for replacement.  (And, yes, there were a couple of accidents involving MLCCs.)

I've been using a heatgun setting on low and trying to evenly heat just the runner. While the heat helps the adhesive release, I worry about these super tiny 0201's (or whatever) accidentally re-flowing. I'm guessing the surface tension would keep it in place.....but who knows?

My technique has been to use a plastic pry tool to get under an edge, and then wand the heatgun back and forth over the runner, all the while applying a lifting force. The runner does sometimes "pop" violently off the board.

I have noticed that the adhesive tape is both still sticky and flexible.....not hard.... and that I've been able to, in 95% of the cases lift the adhesive tape AND the plastic runner off fully intact.

Am I doing this right?

Sounds like your cards are newer if the foam tape was still pliable.  Most of mine were hard as a rock and crumbled when I tried to remove them.

I think if you're heating them enough to reflow the solder, you're applying too much heat!  The plastic runners would probably melt first.

A soft nylon tool works well if you have to do scraping.  I read that some people cut up an old credit card.

[keitheevblog]

Mark,

Of your 25 cards, what cards and years of manufacture are you dealing with?

I have about 10-12 cards, and I've inspected about half of them. The serial numbers are US39xxxx, US40xxxx, and US42xxxx indicating 1999-2002. 16715As, 16717As, 16752A.

I also have MY45's -- Malaysia 2005, 16951s and 16910As.

Is the corrosion you're talking about very visible? I've seen your images posted, and the god-awful ones from Alexandre, but the majority of what I've seen with real problems look like really dirty, dingy, and I'm not surprised by the fact that there's corrosion on these boards.

But mine look nothing like that. They are clean. Now I use my 3M/SDS 497AJN ESD-safe vacuum on things, and IPA to keep them clean. I use a grounded wrist-strap at all times.

I'm not judging or humblebragging about the state of the boards here, I'm truly worried that I'm missing some obvious signs. Now I did see a tiny bit of corrosion on my stacking connector pins which I used deoxit and IPA to clean up.

I'm using 10x magnification to look at the traces underneath the runners and I'm looking very carefully. I just don't see anything.

So I'm trying to figure out why you're seeing it about 100% of the time, and I'm seeing it, well, very little. The most I've seen (and I'm not discounting this) is perhaps a light green surface-level tinge on the vias. And maybe that's worse than the traces!

Thanks,
Keith

[MarkL]

Hi Keith,

The vast majority of my cards are US made.

16740A, 16741A, 16750A, 16751A cards are from 2000 to 2002, all US.

16534A cards are from 1995 to 1999, all US.

16753A, 16754A, 16755A are all from 2002, all Malaysia.

I recently acquired a 16712A card from 1999 (US made) and it is extremely clean, apart from a few dots of green corrosion on some vias.  It tested good and, to be honest, I'm not rushing to remove the runners on this one.

Generally speaking, my cards are all around 20 years old and 100% of them have (or had) corrosion to some level.  The rate of whatever is leaching out of the adhesive is probably related to their environment.  The boards I had go bad on me were in a clean, cool, dry area and were not powered on continuously.  The dust had been cleaned off, but runners not removed at that point.

The only card I have without corrosion is a 16720A pattern generator.  It doesn't have runners on the bottom because it doesn't have any components on the bottom.

I don't have any 169xx cards to compare.

I know that the corrosion has wicked into some of the vias on some boards because it has shown up on the other side in a couple of cases.  And those boards don't work, no surprise.  With no schematics, they are likely scrap at this point.  It's a shame that a 10 cent strip of foam tape is systematically destroying this impressive technology.

[keitheevblog]

Thank you for the detailed reply.

It's a shame that a 10 cent strip of foam tape is systematically destroying this impressive technology.

I couldn't agree more. It is a shame. Of course it's the weakest link in the fence idea. Even if they did a superb job in PCB layout, most of the component choices, good SQA on the software, etc etc --- After 20 years+, which is well passed the original window of support, anything is likely to break down.

I keep checking out my various modules. The 16715A which was misbehaving as a master in some cases, I removed the runners. All the areas underneath the runners were clear.

HOWEVER, I think I finally spotted my first piece of corrosion.

Is this it? Note this wasn't under a runner but bottom right(looking at module right-side up) edge.

I've made a couple passes with deoxit and IPA. I admit having a hard time capturing this image, depth of field, light, etc.

Thanks,
Keith

[NickAmes]

That looks more like delamination on an internal layer rather than corrosion.

[MarkL]

Hi Keith,

I think Nick is right that it's not corrosion.  The corrosion is crusty green on the surface, and when it gets underneath the soldermask it turns the copper grayish brown.

Below are a few examples.

The first two are the practically untouched 16712A that I mentioned.  I had to look hard to find anything, but it's there.

The last two are a 16534A card that I recently received.  This one is in terrible condition and fails self-test, and I was going to use it for a thread on how to repair these cards.  There is severe corrosion to traces, and on this one the metal back plate for the hybrid ADC on top has also been thoroughly attacked (it's not just copper that it likes).

EDIT: Hmmm.... seems the photos came out in the opposite order I posted them.  I'm sure you get the idea.

[keitheevblog]

Thanks much for the sample images.

Yeah, I've been pretty lucky --- nothing even close. It's nice to see images of the exact type here.

The key feature looks like almost neon green crusty deposits.....

I'm going to pull my 16900 modules and check those too. My 167xx modules are all clean the best I can tell.

Thanks

[keitheevblog]

Ok no photos yet because I was too tired to remember to take them.

However.

The 16951A I pulled last night has a considerably different designed runner. First, the runners are easily 1/3 to 1/4 the overall width --- very narrow in comparison. So instead of 6.5mm wide, they are something like 2mm wide.

Next, instead of being about 5" long, they are closer to 3-4" long.

First impression is that they sit a touch more proud....that is, a little taller. And more skinny. And they are less long. They take up MUCH less surface area as a whole. It also seemed to be that their placement was better --- given that they are smaller, are located in more "don't care" areas crossing fewer traces.

It does appear that they might have known of an issue and/or might have just been trying to conserve board space....

This module is from 2006 and looked pretty clean to me. Some surface dust. No visible corrosion that I can see.

Just another data point to put in our caps.

Thanks

[MarkL]

The 16753A/54A/55A cards that I have all have the skinny runners too.  They're 3mm wide, and as you point out on your 16951A, are less in number.

They have less corrosion than my other cards, but it's still there.

While less in number, the runners cross several areas of dense escape vias underneath a couple of the BGAs.  Maybe it's same on the 16951A.  When I took the runners off, many of the annular rings on the vias were partially or completely eaten away and corrosion was down some of the holes.  I couldn't believe the card still worked.  Year 2002 serial number.

Out of the 5 16753A/54A/55A cards I have, only 2 of them work.  The picture I posted earlier in this thread:

  https://www.eevblog.com/forum/repair/series-defect-on-agilent-167xx-boards/msg1353955/#msg1353955

was a repair attempt on a 16755A card.  Also from 2002.

With any luck maybe they figured it out by 2006.

[keitheevblog]

ok fellas, I've got some new information. Been on a warpath for days trying to find the right replacement adhesive tape. I think I've got a good start:

Don't try to use VHB acrylic adhesive tapes on silicone conformal coating. 3M application engineers told me that specifically it won't bond. 3M 830 rubber adhesive 442KW should work here, though. It has a nice feature too "Provides clean removability from many surfaces"

While 3M says you should always test the tape for suitability for your specifically application, they think there's a good chance that the 5952 series, specifically the 5915, should work. You cannot, however, easily remove this tape. It's designed to be permanent, although they publish a PDF with removal instructions.

While anything is possible, the application engineer said that it likely was not the adhesive that leached or had some bad interaction with the substrate/adherend. They said they've got "decades of the various formulations in hundreds of thousands of applications" and they thought it would be common knowledge (amongst his peers) and/or covered specifically as a warning if it was a problem. Take that for what it's worth.

He also said extensive tests with this tape in electronic applications have NOT been done.

I'm inclined to think that there was something corrosive on the board that then got trapped underneath the adhesive at the time of assembly. It's also possible that poor environmental storage conditions exacerbated the problem -- perhaps the adhesive/runner attracting or trapping some bit of water on a stored on the side board.

I think this answer is good enough for me. I will say that shearing off a component is definitely going to happen, so I'm going to replace the runners soon. I'll use just enough tape strategically placed to avoid high density trace/via areas.

[benoar]

Hello everyone, new 167xx owner here and new participant to the forum.

Just got a broken 16750A that I am trying to fix: spotted a broken delay line before buying, did not care much about losing 1/4 of the inputs, but when it arrived it was also missing a ferrite, so no testing possible right now. I also looked at the corrosion and… it is quite bad. For example, attached is a photo of a broken trace by corrosion (tested, no continuity), and another one of the PCB after removing a runner, which peeled the solder mask on some areas. You can also see R519 badly corroded: it came off while removing the runner, too.

I cleaned it with IPA and then tried to tin the traces to protect them. but got more mishap while at it: ripped off a pad (test point) presumably because I was too quick and maybe forgot to dry the IPA, or my iron was too hot (370°C; went to 280°C and still working fine with SnPb); it “popped”, but I continued a bit and tore it off (I was thinking it was maybe because of some discharge because my shitty wallwart-supplied hobbyist Weller leaks current like hell; my ESD wristband tingles when it heats up sometimes…). Also accidentally cut another trace somewhere else with my knife when removing adhesive. Then injured myself with the knife and stopped for the moment. Another photo from the result of peeling the runner with a knife: solder mask is cut off in a lot of small places, not very nice. Maybe I should go the heat-gun way.

I will report if I get anywhere with that.

[TK]

There lots of vias that are corroded and they internal traces could be damaged, making it extremely difficult to repair this boards

[keitheevblog]

While I know this doesn't help any of you that already have corrosion on your boards, I've recently done some runner replacements and wanted to share an update.

• I'm silicone-conformally coating any potentially problem area underneath the removed runners, after cleaning, for future protection.
• I decided on 3M 442KW adhesive tape. I really like this tape, working beautifully. Don't order from digi-key, their stock is old. If you're in US, order from ULINE.
• I do think that being proactive with these is the prudent approach, but there is a risk during runner removal. Take your time.

My updated blog post is https://www.techtravels.org/2019/12/corrosion-near-underneath-hp-logic-analyzer-module-plastic-runners/.

Hope this helps!

[nikodem]

Hi!

I seem to have the same problem. I've recently got a faulty 16750A, that when self-tested returned the following:



Unfortunately I don't know how to make any more sophisticated tests. If anyone could point me to a simplish tutorial about that or give some other info I would be very glad .

And I have noticed the green globs of corrosion sticking from underneath the plastic sliders. I have removed some of them, and the sticky tape is totally dry.

So, my questions is - is it possible, that the self-test failures are due to the corroded tracks? Or is it obvious, that the reason is totally different? I'm asking this, before I spend few hours cleaning the board with IPA and/or acetone from this crap  .

[keitheevblog]

Hey there,

Most of the tests that you're running via that GUI interface are the same tests. What you need to do is telnet into the box(Username:root, pw "uh,uhuh", access the shell, and run the pv command. You can hit ? for help, and most of it is self explanatory. You either select a specific module (or have it execute against all of them), and then pick which tests (or all of them) to run.

There are a couple ways to increase the verbosity of the debugging output.

Are you far enough along to know how to do those steps above?

EDIT: And yes, there's a VERY good chance that the test failures are directly related to the corrosion.

Keith

[nikodem]

Hey there,

Most of the tests that you're running via that GUI interface are the same tests. What you need to do is telnet into the box(Username:root, pw "uh,uhuh", access the shell, and run the pv command. You can hit ? for help, and most of it is self explanatory. You either select a specific module (or have it execute against all of them), and then pick which tests (or all of them) to run.

There are a couple ways to increase the verbosity of the debugging output.

Are you far enough along to know how to do those steps above?

EDIT: And yes, there's a VERY good chance that the test failures are directly related to the corrosion.

Keith

Hi Keith!

Thank you for your quick reply. I've connected the device to my networked and connected to it via telnet. I've run the pv without a problem. I will post the test results here as soon, as I finish the tests  .

What I'm more curious is if it is possible to pinpoint the broken trace or at least find a region, where it might be, based on the failed test or info from the pv. For example - my card failed the comparators test (as shown in the photo) and while cleaning the board from the black goo I've found, that two traces from the DAC to the comparators are broken (and I hope that this is the case with them).

[keitheevblog]

Hey,

Yeah it's certainly possible, but I don't think a guide exists to help you identify the traces or regions easily.

I'd suggest looking at the service guides and manuals that describe the tests. There's about a paragraph of text per test that describes what the test does. There's also a general theory of operation section that describes roughly how the LAs work. You can use those, in combination with looking at the board, to help you figure it out.

Some tests are more useful than others. For some of the memory tests, for instance, they will identify which chip number is failing a test. That chip number isn't specific as a PCB silkscreen Unit Designator (or whatever the Uxx's are called) --- so you have to count chips to figure it out.

I can't easily look this up at the moment, but turning on enhanced debugging can help. There's switches like "d=9,p=9,r=9" that adjust the verbosity of the debug output. I don't think we've found a guide that describes what they do, but I think we assume 0-9, and I think I've done 255 before because why not? Look back through the previous threads here where they are discussed. You can see them at the command line help.

Sometimes, you can see a particular problem that matches to a trace. For instance if bit6 is always 0, when it should be a 1, then that trace could be cut. If things are intermittent, it's less clear. Now which trace is bit6? It's mostly trial and error. With these custom chips, finding datasheets isn't always possible. But things are logically laid out on the PCB, usually. So if it's an 8-bit bus, and you know it's bit6, then count 6 from the left, and 6 from the right.....it might be one of those!

You need to be able to probe the boards live, while they are plugged in. In order to confirm the right voltages/signals when the board is being tested/used. MarkL on these forums is a super smart guy --- he has techniques like installing the board in the bottom slot, flipping the chassis over, taking off the bottom cover, then probing the board.

For full disclosure, while I know the approximate technique on how to fix this stuff, I've never actually done it. There are physical challenges (access to the board while it's powered up), which also can require small pigtails to be soldered to the chips.... This stuff being SMD can be challenging to avoid shorts. Or you need to use microprobes (really small fine clips for legs of chips) which can be expensive.

I attempted repairing a couple bad boards, but it was pretty obvious it's beyond me -- partially in terms of lack of experience and skill, but mostly in terms of patience and willingness. You might be in a different place with all of that, obviously.

I hope this helps
Keith

[nikodem]

Ok, so I've done some more detailed tests over the telnet, and my board failed the following tests:

• anlyBusTest
• clksTest
• measAnlyBusTiming
• cmpTest
• ZoomChipSelTest

Where I can find any info about this tests? I have a service manual from Keysight page, but there is nothing valuable about the these - they just say, thet there is a particular test. What I would want to know, if, based on the above failed tests, do you think the board is worth trying to repair it? I mean - it's fun and all but if this means e.g. an 100% dead important IC then it is not fun any more to waste a lot of time.

So - any ideas about this failed tests? 

[MarkL]

Keith definitely has all the magic incantations for the extra debugging tricks.

Despite my obsession with trying to fix these cards, my success rate is only about 1 out of 3.  It's so unfortunate that "pv" and its capabilities is completely undocumented even now when these products haven't been sold for 20 years.

If you haven't found it yet, there's another thread that may be of some help where the DAC and comparators are discussed:

  https://www.eevblog.com/forum/testgear/agilent-16717a-comparator-and-zoomchipseltest-failures/msg3091674/#msg3091674

Even though it's not specifically your card model, it still uses the same DAC and comparators.  Agilent tried to re-use as much of the front-end design (including timing zoom) between the various modules as possible.

[MarkL]

On your latest post, try turning on some of the more detailed debugging output, as Keith suggested, and just run that one test from pv.

Some of the tests will print out bit patterns and other things that it's unhappy about which may be able to guide to you tracks that are damaged.  Most of the test routines name the specific chips where the test is failing, but it doesn't mean it's the chip.

Not that I'm saying it's not a bad chip, but of the boards I've repaired, it never required a chip swap (unless you count the ones I blew up by accident myself, but those doesn't count).