As has already been said, the layout looks questionable.
Do you have some EM simulation software like ADS, HFSS, Clarity, or microwave office? If so, it'd be an good exercise to simulate your layout to see if it matches your measurement. In addition, you may want to download spice/s-parameter models for the particular components you are using (which will include some parasitics).
Check that the manufacturer's website to make sure all components have a SRF > 1.5 GHz. NP0 capacitors <100 pF should be fine for these frequencies (in general).
It might not quite be necessary at 600 MHz, but I'd suggest going with a 4-layer board stackup, with a controlled impedances. This is pretty cheap from the likes of JLCPCB, but you may need to order from USA manufactures? (I personally like Sierra Circuits).
The GND pads of the components should be directly on the top GND plane, with vias close by. The long tracks to a GND via will add too much inductance.
Once you get into microwave frequencies, you need small component sizes. For >1 GHz, I'd only use 0402 or 0201 (imperial SMD sizes). Perhaps with 500 MHz, 0603 would be fine. The larger packages will have more parasitic series inductance. A 0.5 pF capacitor seems a bit small for 500 MHz... I think your layout will introduce parasitics of comparable magnitude.
JLCPCB has some example layout suggestions. Most of them are microstrip with the GND plane far away from the transmission line, but also CPW is OK to use (assuming you control the transmission line impedance correctly).
So... how to fix your current board? You might be able to improve the GND plane by soldering down some copper tape... might work, might not.
Also, it looks like some of your SMA connector's have non-soldered GND pads? You should solder all 4 GND of each connector (especially since there are not vias close to the connectors)