What about using a standard 4 layer board with the middle 2 being identical? The two inner layers might then look electrically the same as a single middle layer of 3.
I see no reason why that would not work. It may have an impact of efficiency as more of the fields are within the lossy FR4. Also i'm not clear how I would add the connector.
I don't think the FR4 between two middle layers(2,3) would have any effect because if they are identical there should be no voltage between the layers. It wouldn't be difficult or expensive to test a prototype.
There could be benefit of using the 2 inner layers. They will add thickness which will add a 3D element to an otherwise planar antenna. This might affect the antenna beam pattern.
Connection to layers 2&3 could be achieved by using a pattern of vias to connect 1&2 ,3&4. This could be done with blind vias, or more cheaply with full through vias. The connector could then be soldered to layers 1&4.
I don't think there are many reflections heading back into the antenna. This would show up in the SWR measurements.
I agree.
Your suggestion sounds like creating a soft transition between the Er of the substrate and air. It would be an interesting thing to test if I had more time.
I did look at creating a more aggressive lens using a saw-tooth like pattern(fresnel lens). It does on paper seem like it might work, but becomes a bit ugly. In fact as the FR4 represents only a thin 2D slice of the propagating field, it seems likely that any changes will only have a very minor effect.
I don't think that the edge shape of the board will have a significant effect because the % of radiated power that reaches the edge of the board (as opposed to the power in free space) will be small. I don't think a fresnel lens would work at this scale because the lens elements would be too small compared to the wavelengths. I think it would be more likely to act as a diffraction grating.
Your measured data shows that the antenna works. Any other features/modifications are likely to provide small incremental improvements.
My old Professor used to say if it isn't measured it isn't true.
The problem with my suggestions is that they are untested, not quantified and therefore largely worthless speculation.