But I was cherry picking parameters and my math isn't rock solid either. I can imagine there are legit measurement cases where it is a problem.
On a second thought, it comes down to how the noise can be modeled, and how the VNA measures. Consider the power waves \(b_2\) and \(a_1\) the VNA actually measures, say, in a transmission measurement, where \(S_{21}=b_2/a_1\). Let's assume for the moment that we can model the noise contribution of the source by an additive stochastic process \(X(t)\).
Now there are several questions. For example, does the VNA do
$$
\frac{b_2+X(t+\tau)}{a_1+X(t)},
$$
where \(\tau\) is the time delay of the DUT, or is some averaging applied, say over a time interval of length \(T\), and if yes, does the VNA do
$$
\Bigl\langle\frac{b_2+X(t+\tau)}{a_1+X(t)}\Bigl\rangle_T,
$$
or
$$
\frac{\langle b_2+X(t+\tau)\rangle_T}{\langle a_1+X(t)\rangle_T},
$$
where \(\langle\cdot\rangle_T\) means the linear average over \(T\), i.e., \(\langle f(t)\rangle_T=\frac1T\int_0^Tf(s)\,\mathord{\rm d}s\). The measurements in the \(b_2\) and \(a_1\) channel could be taken at different times, which means that \(\tau\) would include this time offset as well.
Also, there is the vector error correction, which depends on more measurements than \(b_2\) and \(a_1\) alone, as was already remarked above.
And I'm not sure if the noise contribution at the power wave level can actually be modeled by an additive stochastic process, and which autocorrelation function or power spectral density it would have. In a real VNA the \(b_2\) and \(a_1\) are determined by IQ mixing, where the phase is determined from \(\mathop{\rm arctg}(q(t)/i(t))\), and the magnitude from \(\sqrt{i^2(t)+q^2(t)}\), from which the complex power wave amplitude is calculated. So it is not obvious what the power spectral density of \(X(t)\) would be. That would need a more detailed investigation.
The conclusion for now is that I'm not sure how much compensation of the source noise actually takes place in a real VNA and how source noise can be modeled. This seems to be more complicated than I thought at first.