Products > Test Equipment
Agilent 16717A Comparator and ZoomChipSelTest failures
<< < (5/6) > >>
mbalmer:

--- Quote from: MarkL on September 09, 2022, 04:26:36 pm ---Does this board work normally?  Does changing the logic thresholds work as expected?  I'm asking becasue we could be looking at a failure of the self-test mode and not an operational failure.

Whichever the case, I think the first suspects are whatever's in common to ALL channels.  There's just too many errors.

During cmpTest, the DAC output values are moved around to create different logic levels and clock patterns for Chip 8 and Chip 9, and those levels are captured in acquisition memory.  pv checks each input bit against what should have been received and that's what those errors are all about.  ("B" == bad bit).

One interesting clue is that all bits on all pods are always bad when there should be no activity or transitions on the ASIC input lines (Cal Clk No Act.).

The posts above from June 2020 show what should be happening on one of the comparators (U69).  A quick probe on U69 with cmpTest looping could be enlightening.

Since the DAC (U34) is in common to all comparators, there could be a problem with the DAC reference voltages (U44 and U45), or maybe the DAC CPU interface (data lines/chip select) have a problem.

There's also a test enable pin that's common to all the comparators (pin 18).  I would make sure that it's showing sane TTL voltage levels and is going low when cmpTest is running.  If it's flapping around because it's trace is severed, that might explain this.

I think we should set aside the zoomChipSelTest for the moment.  As pointed out before, I think the comparator failures reported first are probably the cause.

--- End quote ---

I will have the opportunity to test some of this out tomorrow. Being a band director is sometimes rather involved, time-wise. :D
mbalmer:
Just to brink MarkL up to speed on my TERRIBLE luck with these things:


--- Quote from: mbalmer on September 11, 2022, 02:15:44 am ---I have five boards. Each module fails a different set of tests, although there are some that are disturbingly common.

Board A fails the comparators and ZoomChipSel tests.

Board B fails comparators and calibration.

Board C fails VRAM (both serial and parallel) and comparators. 

Board D fails VRAM, all Zoom chip tests, and one of the inter-board tests.

Board E fails every test except for chip registers, comparators, backplane clocks, the ICR test, and the arming test.

So it seems that while the comparators test is a point of failure, it’s definitely not the only one.

--- End quote ---

I received a sixth and seventh 16717A board today. I plugged what we'll call Board F into the mainframe as the only board installed, and immediately received an error upon bootup that said that the module failed its' high-speed system clock test, and it claimed that the frame may need service.

However, the board passes all tests with x modtests from within pv.

I then re-ran all tests by setting d=9 (but not r=9).

As it ran the test, once it reached the vramCellTest, it flooded the screen with information, mostly clock numbers and hexadecimal codes which I'm guessing relate to specific VRAM cells, the data sent and the data read back. It sat there counting down clock codes from over 2,000,000 for a good solid 30 minutes before it moved on to the next test. After letting it run, all the tests passed on debug level 9.

I pulled Board F out, and replaced it with Board G. The frame powered up and got to the Workspace system without throwing any errors, so I logged into it via telnet and then ran a suite of module tests through pv.

Sure enough, it fails three tests: cmpTest, calTest, and zoomAcqTest.

This one fails cmpTest in an unusual way, however:

 
--- Code: ---  Check POD1 Thresholds:
    Slot E, Chip 9: . ........ ........  . ........ ........  Cal Clk No Act.
    Slot E, Chip 8: . ........ ........  . ........ ........  Cal Clk No Act.
    Slot E, Chip 9: . ........ ........  . ........ ........  Cal Clk Levels
    Slot E, Chip 8: . ........ ........  . ........ ........  Cal Clk Levels
    Slot E, Chip 9: . ........ ........  B BBBBBBBB BBBBBBBB  Cal Clk Activity
    Slot E, Chip 8: . ........ ........  . ........ ........  Cal Clk Activity
  Check POD2 Thresholds:
    Slot E, Chip 9: . ........ ........  . ........ ........  Cal Clk No Act.
    Slot E, Chip 8: . ........ ........  . ........ ........  Cal Clk No Act.
    Slot E, Chip 9: . ........ ........  . ........ ........  Cal Clk Levels
    Slot E, Chip 8: . ........ ........  . ........ ........  Cal Clk Levels
    Slot E, Chip 9: B BBBBBBBB BBBBBBBB  . ........ ........  Cal Clk Activity
    Slot E, Chip 8: . ........ ........  . ........ ........  Cal Clk Activity
  Check POD3 Thresholds:
    Slot E, Chip 9: . ........ ........  . ........ ........  Cal Clk No Act.
    Slot E, Chip 8: . ........ ........  . ........ ........  Cal Clk No Act.
    Slot E, Chip 9: . ........ ........  . ........ ........  Cal Clk Levels
    Slot E, Chip 8: . ........ ........  . ........ ........  Cal Clk Levels
    Slot E, Chip 9: . ........ ........  . ........ ........  Cal Clk Activity
    Slot E, Chip 8: . ........ ........  B BBBBBBBB BBBBBBBB  Cal Clk Activity
  Check POD4 Thresholds:
    Slot E, Chip 9: . ........ ........  . ........ ........  Cal Clk No Act.
    Slot E, Chip 8: . ........ ........  . ........ ........  Cal Clk No Act.
    Slot E, Chip 9: . ........ ........  . ........ ........  Cal Clk Levels
    Slot E, Chip 8: . ........ ........  . ........ ........  Cal Clk Levels
    Slot E, Chip 9: . ........ ........  . ........ ........  Cal Clk Activity
    Slot E, Chip 8: B BBBBBBBB BBBBBBBB  . ........ ........  Cal Clk Activity
> Slot E: Comparator Test Failed!
--- End code ---


I'm about ready to give up on this boat anchor. This is potentially only one out of seven boards I've received that has successfully passed all of its tests, and even then, the mainframe doesn't fully like Board F since it throws a boot-time error about the system high-speed clock.


mbalmer:
Spoken in the manner of Jean-Luc Picard:

Repairs log, HP 16702B, calendar date September 19, 2022.

Board F is, thus far, the only logic analysis board to fully work out-of-the-box and not have issues. I have taken a hot-air gun to the board to gently remove the plastic runners, and they came off remarkably easily. The adhesive strips were supple and not crumbly, and clearly were in good shape. My estimation is that if there is any corrosion on this board, it's minimal at best.

Board F was plugged into Slot C and retested, and the system did not complain about a high-speed clock malfunction.

A 16755D module arrived yesterday and I have not had the opportunity to plug it in and look at it. It may be functional enough that I can use it as a second set of channels.

I've acquired a small set of the Mechano SMD lead grabbers that I can use to get in tightly to the pins on one of the boards. I will be using them, along with a board extension I designed to fully extend the board under test out of the mainframe so that signals can be safely probed on both sides of the unit without risking shorting it out against the case. This extension should theoretically also work for any 165xx, 167xx, or 169xx-series modules as well, so if anyone decides they'd like one, I can offer up the gerbers once I dial in the exact measurements (I was a little bit too wide from side-to-side, but everything else looks good.

Probing and investigation continues.
MarkL:
On that card with the sparse failures, I would check that the comparators are receiving the test clock.  The failed bits are all in the positions where there should be a clock detected (when the DAC is set to 0V for that pod).  All the other bits I think are showing not failed because they are not having any transitions (no clock), which is correct.

Start cmpTest running in a loop and take a look at the ECL clock before it goes into an RC network near the comparators.  See comparator_ecl_clock.png below for the probe point.

You can also verify the enable signal for the clock signal near U74, test_clk_enable.png.

Below is also a scope capture where you can see the clock on Ch1 (yellow) and the enable on Ch3 (blue).  Most of the time the test runs 4ms or so, but it can stretch longer (probably dependent on other housekeeping the system is busy doing since this is not a real-time test).

Also below is a script I've been working on to loop the pv tests.  You can give it a try and let me know if it works ok for you.  I call it "pvl", short for "pv loop".  It's a little bit hack-y in places, but the idea was not to pile on a bunch of prerequisites to run it.  It only needs the stock shell.

The enable signal runs down the edge of the board to the FPGA near the edge connector and also passes directly under one of the runners.  If the enable signal is not showing up, I would check the continuity of that trace from start via to end via. And my general recommendation is to do such a check on ALL the traces under or near the runners.  The corrosion can get under the solder mask and break the trace and it's not always visible.


--- Code: ---#!/bin/sh
# /bin/sh is the POSIX shell on HP-UX.  man sh-posix

# This script sends commands to pv to run a module test on one or
# more slots.

# shortened prog name
#
myname="${0##*/}"


# -------------------- Parse args --------------------
#
opt_w=30

usage()
{
  echo "
usage: $0 [OPTIONS] [<SLOTS> [<TEST>]]

  SLOTS  is a group of letters indicating which slots to test.  Multiple
         slots running the same test can be used to compare a broken and
         working modules.  If the word \"help\" is given, the general help
         will be printed which may have extra information at the bottom
         because of the non-default mode levels being set.

  TEST   is the test to be run.  If TEST is not given, the list
         of possible tests will be printed for the given slot(s).

  If no arguments are given, pv will be run in interactive mode
  with the result, debug, and mode levels set.

Options:
  -wN  run the idle loop N times between commands (60 is approx 1 sec).
       Idle loop defaults to $opt_w if this option is not specified.

Examples:
  Run cmpTest on slots e and b continuously with a pause after each
  set of test:
 
    $0 eb cmpTest
" 1>&2

  exit 1

  #  -sN  sleep for N seconds between commands (overrides -w)
}

while getopts "w:?" opt
do
  case "$opt"
  in
    w) [ "$OPTARG" ] && opt_w="$OPTARG" ;;
    s) opt_s="$OPTARG" ;;
    '?') usage ;;
  esac
done
shift $(($OPTIND-1))

slots=$1
test=$2


# -------------------- Funcs --------------------

waste_time() {
  # Do a slow-ish operation $1 number of times.  This is in place of a
  # fractional sleep command, which HP-UX doesn't have without getting
  # into compiling a program.
  #
  # 30 iterations is about 0.5 seconds on a 16702B.
  #
  local i=$1
  while [ $i -gt 0 ]; do
    ls / > /dev/null
    let i=i-1
  done
}

pv_cmd() {
  # Send command $1 to pv and wait for the marker on the output.
  # Running a system command from pv causes it to flush stdout.
  #
  print -p "$1"
  print -p "! echo %%MARKER%%"
  while read -p; do
    [[ "$REPLY" = *%%MARKER%%* ]] && break
    echo "$REPLY"
  done
}

# -------------------- Main --------------------

# Turn on all pv debug options via env variables (undocumented feature).
# Upping the mode level causes additional hints to appear at the bottom
# of the help output.  A result level greater than 9 shows additional
# information, despite that help says 9 is the max needed.
#
export PVRESULTLEVEL=10
export PVDEBUGLEVEL=9
export PVMODELEVEL=1

# There is odd behavior from pv that prevents calling it from being
# generalized.  It may be closing and re-opening some of the file
# descriptors, but I'm not sure.


(( $# == 0 )) && { pv; exit; }
[ "$slots" = "help" ] && { pv -c help < /dev/null; exit; }

slot_list="$(echo $slots | sed 's/\(.\)/\1 /g')"

if [ -z "$test" ]; then
  cmd_list="$(echo $slots | sed 's/\(.\)/s \1,t label,/g' | tr , '\012')"
  out=$(echo "$cmd_list" | pv)
  echo "$out"
  exit 0
fi

pv 2>&1 |&
PVPROC=$!

while true; do
  for slot in $slot_list; do
    pv_cmd "s $slot\nx $test"
  done
  [ "$opt_w" ] && waste_time $opt_w
done


# The above loop will never exit, but if this script is modified
# for anything else, one of the below can be used to quit pv.
#
#   print -p "q"
#   wait
#
# -or-
#
#   kill $PVPROC
#   wait

exit 0

--- End code ---
mbalmer:

--- Quote from: MarkL on September 25, 2022, 04:31:48 pm ---On that card with the sparse failures, I would check that the comparators are receiving the test clock.  The failed bits are all in the positions where there should be a clock detected (when the DAC is set to 0V for that pod).  All the other bits I think are showing not failed because they are not having any transitions (no clock), which is correct.

Start cmpTest running in a loop and take a look at the ECL clock before it goes into an RC network near the comparators.  See comparator_ecl_clock.png below for the probe point.

You can also verify the enable signal for the clock signal near U74, test_clk_enable.png.

Below is also a scope capture where you can see the clock on Ch1 (yellow) and the enable on Ch3 (blue).  Most of the time the test runs 4ms or so, but it can stretch longer (probably dependent on other housekeeping the system is busy doing since this is not a real-time test).

Also below is a script I've been working on to loop the pv tests.  You can give it a try and let me know if it works ok for you.  I call it "pvl", short for "pv loop".  It's a little bit hack-y in places, but the idea was not to pile on a bunch of prerequisites to run it.  It only needs the stock shell.

The enable signal runs down the edge of the board to the FPGA near the edge connector and also passes directly under one of the runners.  If the enable signal is not showing up, I would check the continuity of that trace from start via to end via. And my general recommendation is to do such a check on ALL the traces under or near the runners.  The corrosion can get under the solder mask and break the trace and it's not always visible.

--- End quote ---

I should have the chance to test this out here in the next week or so. Once I get past our marching contest season, my weekends clear out rather dramatically and I start to feel less like I'm being hounded to accomplish everything all at once.
Navigation
Message Index
Next page
Previous page
There was an error while thanking
Thanking...

Go to full version
Powered by SMFPacks Advanced Attachments Uploader Mod