You should hook it up to an oscilloscope in order to see what's going on.
The output RF amplitude is obtained via a series of attenuators and amplifiers. Here's a rough description.
The finer steps in the output RF level are obtained by varying the dc bias level that is injected in the RF block via the AM input.
If you inject an AM signal it will swing above and below this dc bias level.
This DC bias level needs to be there throughout the entire frequency range of the instrument as it's responsible for keeping a set of PIN diodes polarized.
This signal is controlled digitally and obtained via a DAC and is responsible for -4 db and sub increments.
Aside from this it has a set of 3 fixed, digitally controled attenuators on the RF board that do -20, -10 & -5 db attenuation.
Adding to that, on the attenuator board ( the small can containing the BNC connector) it has a -50, -40 db attenuator plus a +20db amplifier, all of them turned on and off by the digital board.
By using all of these in different combinations the digital board can vary the output signal from -20 to +126db in 0.1 db increments.
Mind you, the +20dB amplifier kicks in above 106db and is hooked in circuit via a relay, you can hear it click when going from 106 db to 107 db.
I have a VP 8194 that had the +20dB amplifier toasted ( among other things) and above 106db no signal was present whatsoever.
My guess is that your unit probably has one of the attenuators broken.
If you need a service manual let me know.