Curious as to what hardware version you guys have - I have a "version 2" (February 2017)
edit: here's what I get when verifying a 3 months old calibration. No issues like the above where values wildly drift.
That's not the issue: you have just checked the calibration accuracy by placing the OSL kit at the Port 1 after calibration, that should be fine even with a 25ohm source impedance if the maths are done properly, that's what the calibration is for (+/- temperature drift). The problem is that having a ~25ohm port 1 impedance will affect the DUT behaviour because of the mismatch, and this can't be corrected by calibration (these things are what makes the professional VNAs expensive). Even measuring a coax cable loss will be error prone because of the S21 ripple generated by the reflections.
Unfortunatelly, you can't check the port 1 impedance without another VNA, because port 2 doesn't have reflection measurement capability (it's not a bidirectional VNA, so only S11 and S21 measurements). But it's clear by looking at the PCB that the design arround port 1 (CN2 connector!) is far from ideal.
You can check the effect of port 1 matching by measuring a couple of meters long coaxial cable. First calibrate the VNA and measure the coax S21 as usual, save the trace. Then place a 10dB attenuator at port 1, calibrate the VNA again and remeasure the same coax. You will see that measurements doesn't match, and the first one will show a huge ripple. You can use the 10dB pad technique to improve this behaviour, but you will reduce the VNA dynamic range by 10dB.
Now imagine trying to adjust or optimize a high Q circuit, like a band pass cavity or duplexer: it will not be possible because it's behaviour will be very different when connected to the VNA. Resonances will change, loss, rejection... You could be able to get the desired response, but it will not work the same way when attached to a 50&50ohm system.
If your VNA behaves differently, please take a photo of the PCB to look for the differences.