Probably not much help, but I had a defective R3765AH as well. Symptom was "random" measurements (essentially just noise) regardless of input to A/B, no RF from Source. YTO was "good" (output frequency was, as far as I could measure, coupled to input voltage), RF block was seriously misbehaving so I first suspected it to be dead, but replacing the RF block with a mixer (which it essentially is: mixer + pga) didn't result in any visible change. Turned out - it was much simpler.
In the end, it was just that all capacitors were dead (truly dead, as in: no measurable capactity). From the outside, they all looked fine. This resulted in SYNTHE board not outputting anything useful (basically the - now unfiltered - power supply ripple caused the power sequencing to reset itself continously), which resulted in the PLL being unlocked all the time; the YTO was never driven with a stable voltage (even with Span=0, which apparently is how you calibrate it). After replacing the capacitors, it's mostly good again, though I'm seeing some excessive noise at some frequencies that I still have to track down.
Can you post pictures from the R3765BH directional coupler?
I don't have any service manual; would be very interested to have one, though.
I also wonder if retrofittinghacking the 8GHz option would be possible as it's essentially just a bypass for the mixer; I could live without the power being adjustable.