Series defect on agilent 167xx boards?

[aeg]

Unfortunately I don't have any extra pod cables, but sometimes they can be repaired depending on the location of the break (assuming you're talking about the coax ribbon cable).

Yeah, the coax ribbon cable. What's the repair technique?

[John_ITIC]

Hi Everyone. I have studied this thread in detail as I'm going through all my 16717A and 16754A LA boards to fix the rail corrosion. The 16717As were quite easy to fix as mostly the usual trace corrosion. One had a via break so took some time.

On one of my 16754As, I have however found a new failure scenario, which is that the SDRAM leads next to the plastic runners are corroding to the point where there are memory errors during the pv 'dataBusTest' test. I discovered this by touching up the solder on the leads/pads to get rid of some brown "crust" and corrosion around the pins. This made the pv 'dataBusTest' fail although the board previously had tested fine all the way up to the timing zoom calibration tests.

When removing the SDRAMs mentioned in the pv dataBusTest (the ones closest to the bottom runner) I noticed the multiple leads were extremely fragile and bent and fell off very easily during cleaning. My theory is that they were almost corroded off and that the thermal shock from the soldering iron caused some pins to crack. I checked the two SDRAMs on the opposite side on the bottom side (next to another runner) and two leads broke off there too.

I have replacement ICs on order and will see if that makes the dataBusTest and subsequent data-related issues go away.
https://www.mouser.com/ProductDetail/Alliance-Memory/MT46V16M16TG-5BMTR

Note: The replacement memory  is capable of higher performance but will behave the same when clocked at the same speed and CL.

Replacement: speed grade 5B: 133MHz to 200 MHz.
Original: speed grade 75: 100 MHz to 167MHz.

Thanks,
/John.

[John_ITIC]

Replacement of the corroded SDRAMs fixed the pv SDRAM related test failures.

I have found yet another failure scenario: shorted out decoupling capacitors under the SDRAM controller IC. Per the attached image, there should be 51 ohm across the resistor. If it is 3-6 ohm or so, then check the nearby decoupling capacitors for shorts. Check resistance from the pads of the resistor that measures a few ohms to the nearby decoupling capacitor pads. If a dead short on both sides (resistor to decap) you know its on the same power rail. Then remove one capacitor at a time until the resistor short goes away. Confirm by measuring resistance of the decoupling capacitor. I had one at 3 ohm and one at 6 ohm. I replaced four on two separate boards and they were all 100 nF 0603's.

Note: If all the SDRAMs in the same side of the board fails 'x dataBusTest', then suspect shorted power rails like above.

[MarkL]

That's an interesting failure on the capacitors.  Thanks for posting the troubleshooting procedure.

How did you figure it out?

Also interesting that there's a lot less of those capacitors under the other ASIC (U126).  Almost all of the channel acquisition circuitry on these boards is duplicated with respect to the left and right, including the resistor you're pointing out.  Just wondering why one side wouldn't need the capacitors.

[John_ITIC]

Regarding the bad capacitors: I found some bad that were not near any plastic runner so it's likely they went bad (a few ohms series resistance) over time for other reasons. I essentially checked every resistor and capacitor for reasonable values and also compared with a good 16754A board.

I am now next dealing with a failing anlyBusTest. The verbose pv test log is attached.

The 16754A service manual says:

"Analyzer Chip Memory Bus Test.
The purpose of this test is to check the Analysis chip memory busses that go between the Analysis chips and the SDRAM controller FPGAs."

So, I assume this test is utilizing the SDRAM controller to read data from the SDRAMs. As the dataBusTest, addrBusTest and other SDRAM related tests have already succeeded, we can rule out bad SDRAMs. The anlyBusTest.txt file shows that the expected data mostly appears some 16 cycles after it is expected. In synchronous logic, this can only be explained by one piece of logic running at a lower clock frequency than intended.

As the clksTest is also failing, I'm suspecting that some board clock is running at a slower clock rate than expected. This could explain why the data appears later than needed. But, I don't yet understand how the clock distribution works on these boards. All I know if from the service guide:

"System Clocks (J/K/L/M/Psync) Test.
The purpose of this test is to verify that the four clocks (J/K/L/M) are functional between the master board and all Analysis chips, and that the two Psync lines (A/B) are functional between the master board's Analysis chips and all Analysis chips in the module. This test verifies that the four clock lines (J/K/L/M) are driven from the master board and can be received by all Analysis chips, and that the Psync lines can be driven by each master chip on the master board and received by all other Analysis chips in the module."

I know, from reading earlier posts, that there had been some prior faults with the clock distribution or PLL chips on these boards. Perhaps that is what is going on. I will re-read to refresh the topic. But, if someone has had a similar issue, I would appreciate receiving some more bread crumbs to follow.

Thanks,
/John

[MarkL]

I agree it looks like a clock failure.  However, I don't think the dead PLL symptoms posted previously resulted in "Analysis Chip Data Bit" errors.  I think in your instance it looks like a data strobe failure between the FPGA (closer to the memory) and the analysis ASIC (closer to the pod connectors).

There's a hefty amount of traces between each FPGA/ASIC pair on the bottom, and I'm assuming you already checked the integrity of those traces.

Note that all the anlyBusTest failures are happening on "Chip 8", which on the 16753A/54A/55A/56A cards is the analysis ASIC associated with Pods 3+4 (U77).  There are no anlyBusTest errors happening on Chip 9, which is the other corresponding ASIC for Pods 1+2 (U76).

So, using this to your advantage, one way to tackle the issue is to compare component values with the corresponding component on the other half of the board.  I've found a number of bad (open) resistors using this method, and I guess capacitors are on the menu now thanks to your latest find.  At first pass, even if you're measuring in-circuit, all you would need to look for is anything that's wildly different from one side to the other.  The actual component value is not so important.

You could also set up a looping anylBusTest and with a scope try to locate anything that looks like a clock on the working Pod 1+2 side, and compare that to the corresponding test point on the broken Pod 3+4 side.  The clock might only be active when moving data.

Your observation that it sometimes eventually gets the test pattern right after a delay may be the result of a clock that's only partially working.  The clock could be running at the wrong logic level due to a bad series or shunt termination resistor, or maybe a bad trace.  Maybe the clock is a differential pair and one leg is dead.

You could also try to force the same error to occur on the working Pod 1+2 side by grounding various traces on that side through a 10R or 20R resistor, while watching the test in a loop with debug on.  I don't know which trace(s) are clock, but I've used this method to home in on non-working individual bits that are reported in the debug output.

I would focus on the "Analyzer Chip Memory Bus Test" failures and not worry about the "System Clocks (J/K/L/M/Psync) Test" for the moment.  The latter test is probably failing as a result of the corrupted path between the FPGA and ASIC (just never say never).

[John_ITIC]

Thanks Mark. Much appreciated.

I re-checked all available traces and components around U77 but could not find anything wrong. I have been in touch with DogP and it looks hopeful that I can get my hands on a few of his extender cable adapter boards so I can do further bench testing with my scope on various live signal paths.

I have four of these 16754A boards that fail the pv comparatorCalTest and dappAddrDataTest too. And I know that these are related to the timing zoom, as the GUI calibration also fails. And I get timing zoom errors when sampling with timing zoom turned on. It will be tricky to resolve with the boards inside the 16700 case so I prefer to get the proper extension cable setup.

I can confirm that the anlyBusTest failures only occurs on pods 3 and 4. As an experiment, I fed 16-bit wide ripple counter data from an Altera dev kit (with 0.1" header LVCMOS outputs) to the pods via a flying leads probe. This test shows that I can actually consistently configure and sample data from all pod channels so I know that all data paths are intact. But channels 3/4 have some odd behavior approximately 50% of the acquisitions:

On Ch 3/4, I'm seeing lots of toggling of the bit state before it settles to the correct state. See attached images:

p515: "Bouncy" transitions on channels 3/4
p516: The same capture but zoomed in such that the "bounces" can be seen more clearly.
p517: A proper capture of channels 1/2. Half of the times, channels 3/4 look this way too.

 I have to think some more about what could cause this behavior.

Thanks,
/John.

[John_ITIC]

I have a few more key pieces of information regarding the "bouncy" transitions:

1) The bouncy phenomenon starts when the test vectors change from all 1's to all 0's. See attached p518.
2) The trigger does not catch this when triggering on D0 positive edge to D0 positive edge less than 80ns (80ns is the D0 period).
3) The fact that all exhaustive SDRAM pv tests pass fine suggests that this is an issue with the U77 analysis IC, not with the SDRAMs or SDRAM controller.

This tells me that this is an issue with the U77 analysis IC retrieving the data from the FPGA controller IC after capture.

This looks suspiciously like ground bounce, suggesting an issue with the U77 power integrity. Perhaps bad decoupling or bulk capacitors.
https://en.wikipedia.org/wiki/Ground_bounce

I'm investigating further...

/John.

[MarkL]

...
I re-checked all available traces and components around U77 but could not find anything wrong. I have been in touch with DogP and it looks hopeful that I can get my hands on a few of his extender cable adapter boards so I can do further bench testing with my scope on various live signal paths.
...

I also have DogP's extenders and they work well, but before that I did the vast majority of my troubleshooting by putting the target card in slot E, turn the chassis upside down, remove the bottom cover, and safely relocate the mouse/keyboard interface card.  You can get to nearly the entire bottom of the card.  You can get a lot left/right comparisons done this way, and if you're looking for a bad trace, it's most likely going to be on the bottom anyway.

And if you're using remote control, you can completely disconnect the mouse/keyboard card.  The OS doesn't care if it's there or not.

[John_ITIC]

I also have DogP's extenders and they work well, but before that I did the vast majority of my troubleshooting by putting the target card in slot E, turn the chassis upside down, remove the bottom cover, and safely relocate the mouse/keyboard interface card.  You can get to nearly the entire bottom of the card.  You can get a lot left/right comparisons done this way, and if you're looking for a bad trace, it's most likely going to be on the bottom anyway.

And if you're using remote control, you can completely disconnect the mouse/keyboard card.  The OS doesn't care if it's there or not.

Thanks Mark. I considered this after reading about your initial setup but since DogP had available extender boards (on the way to me) I'll go for the "proper" bench setup. I'm in no rush to get this system back up and running so will take my time to make my job easier (even if the initial setup takes a little longer) by having the whole board on the bench.

Otherwise, no update on this issue today.

/John.

[John_ITIC]

The update for this weekend is that I assembled DogP's adapter cards and managed to burn up both one of my 16754A cards as well as my 16700B main frame! It was my fault as I missed DogP's comment not to use 80-wire cables as those short together some 6-7 pins on each cable. So I'm putting this in here as a warning to others: Only use 40 pin connectors and 40 pin flat cables.

https://github.com/pdaderko/16702B/blob/main/card_extension/README.md

I looked into the 16700B issue and found that it does not detect any (known good) boards. The back plane does not have any sensitive electronics so I assume its the main interface board that somehow got damaged. I did find a replacement main interface board on Ebay so I picked that up as that would be a quick and easy fix if it resolves the issue.

Now, having learned my lesson, I tested the 16700 extension boards in another 16702A unit with 40 wire flat cables and I can report that they work fine with 21" cables. They are a little too short for my work bench arrangement so I put in an order for very long 52" cables. I think that will work well for the pv self tests as no major data is transferred over the cables. I would assume that actual data capture and upload will stress the cable interface more - I will try that out once the long cables arrive.

Finally, I attached a couple of pictures for the curious...

Thanks.
/John.

[dorkshoei]

I did find a replacement main interface board on Ebay so I picked that up as that would be a quick and easy fix if it resolves the issue.

Thanks for the order though I hate giving 13% to those eBay bloodsuckers.  I have several other boards from the same 16700b for sale.  All working. PM me here.

[DogP]

Ouch!  I guess that's good to confirm my suspicion that damage would occur from the wrong cable... would have preferred to have kept that unconfirmed though. :-/

To track down what likely got damaged... those 80-pin cables internally short pins 2,19,22,24,26,30,40.  Pins 19 and 22 are unused on the extension, leaving pins 2,24,26,30,40.  That would cause backplane pins 2, 20, 22, 26, 36 to short together from one cable, and pins 38, 56, 58, 62, 72 to short together from the other cable.

I'm not aware of the backplane pinout being published anywhere, but simply using a voltmeter may find the obvious culprit (e.g. finding +12V, -5V, etc. which would have likely damaged a data or control signal).  Or maybe a voltage rail got shorted to another, possibly popping a fuse.  No idea if they buffered anything, or if the lines went directly to FPGAs/ASICs/etc. though.

I'd definitely be interested to hear how the long cables work out... another option is popping the bottom cover off the unit like MarkL mentioned, putting the unit on its side, and then using the extension out the opening.  That's how I tested cards on my workbench, shown in this pic: https://github.com/pdaderko/16702B/blob/main/card_extension/in_use.jpg .

DogP

[MarkL]

...
To track down what likely got damaged... those 80-pin cables internally short pins 2,19,22,24,26,30,40.  Pins 19 and 22 are unused on the extension, leaving pins 2,24,26,30,40.  That would cause backplane pins 2, 20, 22, 26, 36 to short together from one cable, and pins 38, 56, 58, 62, 72 to short together from the other cable.

I'm not aware of the backplane pinout being published anywhere, but simply using a voltmeter may find the obvious culprit (e.g. finding +12V, -5V, etc. which would have likely damaged a data or control signal).  Or maybe a voltage rail got shorted to another, possibly popping a fuse.  No idea if they buffered anything, or if the lines went directly to FPGAs/ASICs/etc. though.
...

I've done a little poking at the backplane about a year ago, and was able to take some reasonable guesses at some of the signals by direct measurements and looking at a broken 16531A scope module.  The 16531A is thru-hole, so it was a lot easier to see where various traces were going.

Here's what I have so far.  Any pin where I didn't have a good guess is not listed.  I would think shorting the -5.2V supply did the most damage, since it's also responsible for supplying the power-hungry ECL on the modules.



From service guide, power supply has:
  +5V  -5.2V  -12V  +3.3V  +12V  -12V  -3.3V

Connector: AMP 1-534204-4 (pin 1 has a square pad)

Pin(s)  Signal
------  -----------------------
1-5     +5V
6-8     +3.3V
9-10    +12V
11-12   -12V
13-14   GND
15      D7 bi-directional (D7:D0 based on 16531A DAC)
16      D6 bi-directional
17      D5 bi-directional
18      D4 bi-directional
19      D3 bi-directional
20      D2 bi-directional
21      D1 bi-directional
22      D0 bi-directional
23-24   GND
26      from backplane
27      from backplane
28      from backplane, A1 on DAC
29      from backplane, A0 on DAC
30      from backplane, 138 both decode select C
31      from backplane, 138 both decode select B
32      from backplane, 138 both decode select A
33-34   GND
35      from backplane, R/nW, select which 138 decoder
37      from backplane, 134 both G2A not enable
43-44   GND
53-54   GND
55      100MHz system clock
56      100MHz system not clock
62      from backplane, 138 decode G1
63-67   -3.3V
68-72   -5.2V

[John_ITIC]

Thanks for the order though I hate giving 13% to those eBay bloodsuckers.  I have several other boards from the same 16700b for sale.  All working. PM me here.

PM sent.

[John_ITIC]

That would cause backplane pins 2, 20, 22, 26, 36 to short together from one cable, and pins 38, 56, 58, 62, 72 to short together from the other cable.

I've done a little poking at the backplane about a year ago, and was able to take some reasonable guesses at some of the signals by direct measurements and looking at a broken 16531A scope module.  The 16531A is thru-hole, so it was a lot easier to see where various traces were going.

Here's what I have so far.  Any pin where I didn't have a good guess is not listed.  I would think shorting the -5.2V supply did the most damage, since it's also responsible for supplying the power-hungry ECL on the modules.

Thank you both. I will eventually make an attempt to fix the shorted 16754A. But first, Ill take a stab at my three other 16754A boards that all fail the RC comp and Tap delay calibration. I have one good 16754A board that I want to compare with live on the bench.

So here's a summary of your information:

Pin(s)  Signal
------  -----------------------
1-5     +5V
6-8     +3.3V
9-10    +12V
11-12   -12V
13-14   GND
15      D7 bi-directional (D7:D0 based on 16531A DAC)
16      D6 bi-directional
17      D5 bi-directional
18      D4 bi-directional
19      D3 bi-directional
20      D2 bi-directional   <= shorted to +5V (pin 2)
21      D1 bi-directional
22      D0 bi-directional   <= shorted to +5V (pin 2)
23-24   GND
26      from backplane      <= shorted to +5V (pin 2)
27      from backplane
28      from backplane, A1 on DAC
29      from backplane, A0 on DAC
30      from backplane, 138 both decode select C
31      from backplane, 138 both decode select B
32      from backplane, 138 both decode select A
33-34   GND
35      from backplane, R/nW, select which 138 decoder
36   UNKNOWN            <= shorted to +5V (pin 2)
37      from backplane, 134 both G2A not enable
38      UNKNOWN            <= shorted to -5.2V (pin 72)
43-44   GND
53-54   GND
55      100MHz system clock
56      100MHz system not clock   <= shorted to -5.2V (pin 72)
58      UNKNOWN          <= shorted to -5.2V (pin 72)
62      from backplane, 138 decode G1   <= shorted to -5.2V (pin 72)
63-67   -3.3V
68-72   -5.2V

I also found another two 16754A boards sitting in a stored away 16702A so I will need to fix the rail corrosion on those too.

This will all take a lot of time...

Thanks,
/John.

[dorkshoei]

Thanks for the order though I hate giving 13% to those eBay bloodsuckers.  I have several other boards from the same 16700b for sale.  All working. PM me here.

PM sent.

If anyone else wants parts off the 16700B that I put onto eBay (that John_ITIC bought the interface pcb from). PM me.   I'll happily offer 10% discount for direct sales.

[John_ITIC]

John_ITIC bought the interface pcb

I can report that the replacement main interface board made my burnt out 16700B work again. Thank you!

I also have made another interesting observation:

Having moved 16754A boards between various systems (16700B and 16702A), I noticed that some 16754A boards fail more pv self tests in some 16700 systems than in other 16700 systems.

I also noticed that the some 16754A boards could not handle even 17" long extension cables without starting to fail the dataBusTest, hwMemoryCellTest and some others. However, other 16754A boards worked fine with 40" extension cables. I have not managed to get any 16754A board pass self tests with 52" extension cables, however.

I'm suspecting not correct power integrity on some boards / 16700 combinations can cause memory test failures. The search goes on.

/John.

[dorkshoei]

John_ITIC bought the interface pcb

I can report that the replacement main interface board made my burnt out 16700B work again. Thank you!

Great.

Please post if you get the old board repaired

[MarkL]

FWIW, I've been using DogP's extenders with a 16" cable without any issues.  So far I've only used it with a 16752A card.

I added some connectors in the middle of the extender cables to probe bus power and signals more easily.  The red marks are positive supplies, blue are negative, and black are grounds.  Unused wires in the ribbon are covered with red tape.

The ribbon is also slit in one area, with the wires being grouped into each supply rail.  This allows me to measure the current on each rail by passing the group through a current probe.

Being paranoid, the bus numbering is everywhere.  And given the number of fans in the chassis, I felt a fan blowing over the heatsinks when the card away from home would make it feel less home sick.

[John_ITIC]

The updates for this weekend:

1) I have managed to get the 52" cables to work. I found that there is too much resistance (or inductance) in the few GND leads in the long flat cables. The trick with using longer cables is to use additional lab cables to create additional GND paths between the 16700 system and the board under test on the bench. I used four additional regular lab cables between the 16700 chassis and the 16754A and managed to pass all self-tests of my 16754A board. Note that these boards are designed to get additional grounding via the rear panel. While working on the boards, I have removed this panel so very little ground return path remains via the few pins in the backplane connector.

2) A 16754A board that previously passed all memory tests suddenly failed them again.  See attached p554. pv told me that the the following SDRAM ICs were bad: U91, U93, U95, U97 and U92, U94, U96, U98 as well as U89. These are all the SDRAMs on one of the two SDRAM controller FPGAs/ASICs. I traced this down to a shorted power rail on that SDRAM controller. Just like another board, one of the 100nF decoupling capacitors had developed an internal short (< 4 ohm), which collapsed that power rail. See attached p560b and p560c. The short was in the capacitor most to the bottom in image p560 although that cap looked good. The ones above looked crusty and corroded so I swapped them out too, although they measured okay.

3) I now get an intermittent cmpTest (and vOffsetTest) error on this board. See p561, p563, p564. I'm not sure why this pops up now as I never have seen that before. I will have to review under microscope again to see if I can spot anything unusual. And, I have to look into where the DACs are located...

So, some issues have been resolved but new issues have arrived. Never a dull moment...

Thanks,
/John.

[John_ITIC]

It seems I have a bad comparator. The (attached) pv cmpTest and vOffsetTest outputs told me that there is an unexpected voltage on the 2nd comparator on pod 1. I measured across the input shunt resistors and found one that measured 1.5KOhm across while all the others were 4.38K in-circuit. I removed two of the resistors (both measured 20K out-of-circuit) and I found that the input resistance to ground on the bad channel was 435R rather than 4K like on the neighboring good channel.

I recall that others have reworked the comparators before but I don't think it was posted how to get the heat sink off. Or, what to use to put it back once the board has been reworked.

I may be able to heat it sufficiently with the heat sink attached as I have a rather hefty rework machine, with a bottom pre-heater:
https://www.zeph.com/bgarework_stations_systems_qfn_smd_hot_air_repair.htm

The ZT-7 rework machine has a shroud that completely will contain the heat sink inside. So, the heat will not be able to escape when de-soldering.

I have my burnt out 16754A that I could borrow the comparator from. Or, I could order a new comparator as I believe it was mentioned in this thread that they are still available for purchase. I will go back and re-read to see if I can find that post.

Thanks,
/John.

[Hamster]

i thought the comparitors were the smaller agilent chips by the pod inputs without heatsinks?  I posted a some 16700/16751 cards for sale if anything is useful.

[MarkL]

...
I recall that others have reworked the comparators before but I don't think it was posted how to get the heat sink off. Or, what to use to put it back once the board has been reworked.

I may be able to heat it sufficiently with the heat sink attached as I have a rather hefty rework machine, with a bottom pre-heater:
https://www.zeph.com/bgarework_stations_systems_qfn_smd_hot_air_repair.htm
...

I've done rework on the comparators and have never been able to get the heatsink off.  I just moved the whole thing.  I used a pre-heater, as you are suggesting.  Below is a photo of what's underneath if it helps.

i thought the comparitors were the smaller agilent chips by the pod inputs without heatsinks?  I posted a some 16700/16751 cards for sale if anything is useful.

That's on other cards (40-pin cards).  The 16753A/54A/55A/56A and 16760A (90-pin cards) have a very different front end with differential input comparators.

[John_ITIC]

Thank you all for the help. I would be interested in purchasing a parts 16754A so I can get a replacement comparator as well as a replacement U128 Xilinx XC2V100 FPGA, which is used to implement the bus interface. I have confirmed that mine got damaged when using those 80-lead extension flat cables. U128 gets very hot when the damaged board is powered up. All power regulation otherwise appears fine on that board.

I have done some BGA removal, reballing and re-assembly in the past so there is a good chance I can get the board up and running by swapping out the FPGA. There is no programming involved as the FPGAs configuration image is stored in the PLCC XC18V00 PROM, which is next to the FPGA.

Based on this thread, there should be quite a few piles of 16754A boards where the repair attempts have been given up. Please PM me if anyone is interested in selling me a parts board.

Thanks,
/John.