Here is the strange part: I do not receive any signals OTA when attaching a real AIS antenna. But once I start probing around with a spectrum analyzer, I am able to decode incoming messages for as long as my probe is touching the signal path anywhere on the board. Also, it does not matter if the ground of the probe is connected to my circuit or not. (I am using a rather cheap TinySA handheld spectrum analyzer here.)
I see the following possible issues:
- bad untuned antenna
- bad umatched transmission line
- bad design input RF frontend on your circuit (unmatched transmission lines, reflections and other issues due to bad PCB layout or mistake in RF circuit)
- burned out input port of your device
Since you can receive signals when connect some metal thing to the transmission line, it looks that your input port is not burned out. But it worth to check it. In order to do it, try to connect your signal simulator through RF attenuator to get low signal level, just about 10-20 dB higher than guaranteed device sensitivity. If it works, then port is not burned out.
For other issues, you're needs Vector Network Analyzer in order to test your antenna, transmission line and device input impedance.
Could you show the photo of your connections between device and antenna? It can help to reject simple mistakes.
What I also did was trying a different DC blocking cap: The original circuit wanted a 10pF C0G(NP0) 0603 which I did not have.
that part of your circuit looks very suspicious, it may be the root of cause of your issue.
Could you show it's schematic and the photo of PCB?