Thank you both.
I finally got my unit calibrated! But i'm not sure what made the B display suddenly display the correct 1V voltage, as configured in the 100 KHz mode. I suspect relay K401.
This morning, experimentation showed that the 1V mode output was correctly 1V RMS in the 100 KHz mode BUT the ADC thought it was only some tens of mV, as displayed on the B display. I noticed that relay K401 near the ADC was activated when switching from the 10KHz to 100KHz range. So, I wanted to probe the input to the ADC (U422/AK9201A-VP, pin 19). While probing I observed that no input signal to the ADC was available while in the 100 KHz mode. However, after switching the relay some 10's of times, the signal suddenly appeared and the B display displayed correct voltage. So, I'm suspecting that the K401 relay (TQ2-12V) had dirty contacts. Or some part behaved differently when heated up. Or some cold solder joint behaved differently after the PCB warmed up. I'm not entirely sure but from past experience, bad relay contacts on old instruments is quite common and can have similar effects.
I found a link to a schematic containing the ADC here:
https://www.eevblog.com/forum/testgear/$50-for-schematic-of-hp-4338b-milliohmmeter!/
-Note: For a completely different product but I learned about the ADC pin-out.
With the 100KHz/1V mode working, the CAL program finished without further problems. I quickly tested some inductors and capacitors and the 3330 is working properly. The power on self test now passes okay. I have saved the calibration constants for later use, if needed.
I will check how the unit behaves tomorrow morning after having been being turned off for a long time. I might also order a couple of new relays since there is also one (K301) between the HCUR and HPOT connectors. They are only $3.38 each. If the issue comes back I can at least rule out bad relay contacts.
https://www.digikey.com/product-detail/en/panasonic-electric-works/TQ2-12V/255-1002-5-ND/251774.I also noticed that the 1V output in 100KHz mode is slightly lower than the other frequency modes. Both display B and my external DMM agrees with regards to RMS voltage.
120Hz: 998mV, 1KHz: 1.000V, 10KHz: 995mV, 100KHz: 981mV.
I also attached a full view of the PCB.
One question I have: Will it matter whether I calibrate using a Kelvin clip cable?
open: Leave alligator tips unconnected.
short: Short together alligator tips.
Resistor measurements: Connect between alligator tips.
I then also don't have to create any special jig and put my reference resistors in boxes with BNC connectors.
I plan to use the instruments with these alligator Kelvin clips so i figure the calibration would be more accurate if calibrated with the same cable?
https://www.ebay.com/itm/Kelvin-LCR-Test-Tester-Probe-Lead-Clip-Cable-With-4-BNC-For-LCR-Meter-Terminal/253809323050/John.