I would see what the reference and cal/signal zero look like (N=1,2,3) at TE1. I would use a DC, not AC, signal to test the VDC function. Then I would check the ADC counts output (while not in the diagnostic mode) on both sides of the opto-coupler (see theory of operation, table 6-13 and schematic). I would change the input voltage and observe how the pulse chain changes if you change (eg. invert, or change by a factor 2) the input signal.
Attempting calibration if you suspect the ADC circuit is broken is a bad idea. The power supply was probably not the problem (although switching it off is not a good way to get 0 V, you should short the inputs), but the ADC was feeding the MPU garbage information. After repair, you will need to send it out for calibration or have access to accurate 300mV and 3V sources to bring it within spec. I would probably switch to the 30 VDC range for testing, although it involves more components and is therefore less ideal for testing.
Just getting back to this.
So, putting in a 2.50V DC signal from my FG. Put the unit in Diag for 300V (3V, and 30V both give overflow) DC. Checking TE1 at N 1,2,3,4 all give nothing. zero voltage (this is with my scope lead ground hooked to the Lo input, and probing TE1). TE2 gives the same. I'd almost wonder if a relay is stuck open/closed... or a bad FET, or something.
Interestingly, I AM seeing that signal on K3 and K4 (on both sides). According to the diagnostics table, these should NOT be closed.
Checking 6-13, I'm having a bit of trouble on two points.
1. Where are all these referenced to (guessing digital ground), which should be the shared ground but I'm having a hard time finding a good grounding location. It would appear the pin closest to the 5V digital regulator that goes from the main board, to the digital board, is ground for the digital board, as it's connected to Vss (pin 1) on U8.
2. I can't find U3B, which is supposedly where to measure the A/D counts. I see U3, but U3B is no where to be found (though, I'll fully understand if it's right infront of my face).
Anyway, with all that, here's what I'm seeing:
U8-pin2: 1ms (good)
U8-pin3: 11ms (should be 15ms)
U8-pin6,7,8: 5V, pulse when button pressed.
U8-pin9: 0V (no pulse when buttons pressed)
U8-pin19: lots of pulses from 0 to +5V. Seemed to have some "runts" in there as well (much shorter duration than "normal" pulses).
U14 pin 4: 22ms 5V to 0V pulses. No variation.
U3B: Unknown (can't find it). U3 (incase it's a typo) pin 13 gives a 3.84197Mhz 0V to 5V square.
U8 pin 12: 1.92098Mhz 5V clock.
U8 pin 13: 960.491Khz 5V clock
U8 pin 14: 480.246Khz 5V clock
U8 pin 15: 240.123Khz 5V clock
U8 pin 16: 240.123Khz 5V clock
U10 pins 34 and 35: 2.0032Mhz 5V clock
U10 pin 37: 5V
U5 pin 9: 0V to 5V pulses, 1ms apart. Getting some definite noise, pre-pulses, runts, etc in there.
U5 pin 19: 5V
U5 pin 18: 2Mhz square 0V to 5V.
U5 pin 28: 3.72V
As a note, I had to re-close the "scanner" jumper (since the scanner wasn't attached) to get the A/D counts to work, as well as the 2Mhz square out of U5 to be steady.
I'm not exactly sure the U5 oddity is anything significant since I believe that's just the GPIB driver. But it behaving oddly might be a symptom. Or, maybe what it's doing is normal.
And that's it. not exactly sure what to test next. Nice to get some readings, but a bit frustrated that nothing is jumping out here.
As for measuring on each side of the optocouplers... they're UNDER the digital board... a bit hard to measure. =/
Any thoughts? Thanks for all the help!