The -30dBm cal signal appears to be working correctly. I have used the 8552 IF plugin with other units and it operates normally.
However, it seems I have made a discovery of sorts. I tried inputting a -30dBm signal from my 8640B, and similar to the cal signal it was down near the noise floor. I then started to increase the power level, and suddenly when switching from -10dBm to 0dBm, the peak jumped up by almost 40dB to just a few dB below 0dBm. Interestingly, changing the log reference level seems to behave normally and changes the position on the screen by 1 division, but raising the input attenuator up to -10dB made the signal drop down by 40dB again, so it appear to be somehow related to the IF signal level at the output of the mixer. Perhaps the biasing to some amplifier stage has gone open-circuit, and only turns on once the IF signal level is sufficiently high?
Actually, thinking back on it, when I had tried injecting a 200MHz signal at the IF amplifier, I think I had noticed a similar nonlinearity where the trace jumped by multiple divisions when changing the power level, but at the time I hadn't been paying attention to it and had attributed it to the narrow bandwidth of the IF amplifier.
I definitely want to double-check that, but the SMC adapters are back at work, so it'll have to wait until tomorrow.
Since I had been seeing the 0Hz spur as normal, I had foolishly assumed that the problem had to be before the IF amplifier. However, it would seem that the LO-IF leakage was sufficient to appear normal on the screen.
Stand by for further testing, but it seems that the mixer is working normally. Next step will probably be to check the signals before and after the IF amplifier and see what the deal is with that.