I've done a bit of work on several points. Firstly, the effect of using the SMA-SMA cable. Secondly, looking at the noise pickup in the probe. Finally, looking at the issue where the probe responds differently depending on whether it's clamped or held.
Maybe unsurprisingly, using the SMA-SMA cable shows a fairly significant improvement in impedance matching to equipment with SMA connectors. It removes the need for an BNC-SMA adapter, which I've found in the past could be a fairly large source of error in the match. See for example this S21 measurement using the new cable:
Much cleaner with reduced ripple/reflections.
With regards to the noise pickup. After a bit of toying about, I have confirmed that the issue is relating to picking up noise on the signal input of the op amp. As the op amp isn't unity gain stable, its design gain unfortunately amplifies this noise. The obvious answer is shielding which can be achieved with copper tape around the front of the probe, near where the amplifier input is. There is much less benefit from wrapping any other part of the probe in copper tape, and it is not really recommended as it can damage the sticker on the probe. The copper tape can be grounded to the SMA connectior. Care must be taken to not have the tape any further forward than in the image, otherwise the ground starts to have an affect on the linearity of the probe, and even the probe loading on the measured signal if the tape is pushed much too far forward. Unfortunately, the noise pickup is dramatically improved the further forward the tape is pushed, so a balance must be achieved. With the tape as it is in the photograph, I see an improvement of the noise pickup of around -5dB, with little affect on the frequency response, although this is a little difficult to measure consistently given the nature of the radiators that I was subjecting the probe to. This is still going to be a fair bit of noise, but it is an improvement, with a fairly easy implementation if you do need to reduce the noise level.
As an aside, this is one of the reasons why adding a metal enclosure is not really viable, as the closer the metal is to the signal pin and associated circuitry, the higher the tip capacitance. I think it would be quite challenging to achieve the 0.7pF tip capacitance with a non-insulating enclosure.
Finally with regards to the issue where clamping the probe results in a poor looking frequency response. I've tested this and there is a difference in frequency response depending on whether the probe is clamped or held.
Clamped (not handheld)
Handheld (not clamped)
Basically, I think this is caused by a specific grounding configuration, and the addition of a hand modifying this somehow. The addition of the copper tape shielding doesn't help here as it's not actually modifying the ground connection of the probe. I think I see some improvement by using a more substantial ground connection off the ground pin of the probe, by basically soldering together some of the pins in the pin kit to make a DIY ground blade pin. I need to do some more testing to confirm this. Something I've not tried is to use a soldered ground connection with a copper braid, I think that might make quite a difference. I'll continue working on investigating this issue, however I'm not sure it'll ever be completely 'solved' as different grounding configurations will always have an effect on the probe response. Also on my equipment the effect is much more subtle, which makes looking for improvements more difficult. For now I'd say keep using it handheld, but hopefully if you don't the modification of the probe response would be relatively minor so long as your probe grounding is reasonable.