Makes me feel like a first year EE student again, adding capacitors hoping it'll solve a problem, though with more knowledge about the difference it makes, and apparently it's still a valid approach.

I am near retirement and still feel like a first year student.

How did you improve the decoupling department over there? What did you add where?
A couple of things. There was no attempt what so ever to look at this from anything beyond an empirical test. I am not suggesting these changes will help or hurt the performance. My only goal in adding them was to see if I could detect a change in the performance of the VNA after I had damaged it. It was part of my hunting down the root problem. I just never backed out the changes and in my case noticed an improvement at the higher frequencies.
Lets start with the same schematic for the H4. See attached. This will not match your H4. Note the lack of the DC blocking caps.
I placed a 4.7uF 0603 cer.
across C10, C39, C41, C43. I placed a 10uF 1206 cer across C61 & C62 (doubt these made any difference, I was shotgunning). I placed a TVS on each port to ground, at the connector. The parts I had on-hand are about a half puff. While they did not effect the performance, I would look for a better part. Digikey has some with much lower capacitance.
I'm still surprised I damaged mine and my only goal was not to loose it to ESD a second time. If you go ahead and add those first four caps, I am interested in seeing if you find any difference, and what mixers were installed in your H4.
****
When I state across, I turned the caps on-end and solder them right to the pads along side the original caps. For the TVSs, I scraped a bit of mask and soldered them right to the ground plane, near the SMAs.