Engaging the -20dB input attenuator made no difference. In fact I would have expected that with -20dB on the line gets -20dB down, but no, it was at the same place as before. I don't know if this is right?
Yes, it's right, the input level hasn't changed, just the attenuator.
Mark which is which, and try swapping the Ch A and B input boards and see if the problem follows the board or stays in the slot. If its the slot, it is probably the LO signal from A4.
If its the board, you'll have to put it on an extender and trace signals. Do you have the service manual?
You could check the output of the buffer amp (R115) with another channel to see if its flat. After that, its the filtered outputs of the mixers.
I've only encountered one such problem and it was the LO signal, and affected all three boards. It's still waiting for me to fix it
but that unit has pretty much become a parts donor.
But as I said earlier, if you're just doing freq response and other normal VNA work, I'd just unplug the bad B channel board (or just cap the input and don't use it) and carry on with just the R and A channels. The 3rd channel was an option with the 3577B, and no changes or jumpers are required when its present or not. The software just ignores the missing channel if you don't try to use it. The "A" behaves the same way...