Ok, it looks like I opened up the can of worms in Pandora's box, or whatever mixed metaphor you like.

I am going to try to put an example forward. Say the PLL VCO is 600 MHz, and we want the RF signal to be 10.25 MHz and the LO to be 10.26 MHz. The integer divider for both the RF and LO signal is 58. The fractional part for RF=10.25 MHz is 0.53658 and for the LO=10.26 MHz it is 0.47953. The divided by 58 frequency is 10.34482 MHz and the divided by 59 frequency is 10.16949 MHz. So we expect the RF signal to actually be at a frequency of 10.34482 MHz for 0.46342 of the time and 10.16949 for 0.53658 of the time. Likewise, the LO frequency is at 10.34482 MHz for 0.52047 of the time and 10.16949 MHz for 0.47953 of the time. While some of the pulses have a period of 96.7 ns, and others 98.3 ns, they must be interleaved so that the instantaneous IF beat frequency between RF and LO does not deviate too much. I am not sure how the SI5351A actually interleaves the two sets of divided pulses, perhaps it uses a sigma-delta modulator? Nevertheless, one can clearly see the period of the IF vary on the scope. For a 3.333 kHz IF frequency (3rd harmonic) I see about 2-4 microseconds of jitter. For a 300 microsecond period, that corresponds to 3 microseconds = 3.6 degrees or 0.0628 radians, which suggests an instantaneous frequency variation between 3.300 and 3.367 kHz, but it averages to 3.333 kHz. Because the SNR of a single measurement is approximately 1/(radians) that would be a SNR of about 16 or 12 dB. But if we integrate over many cycles, we can improve this by assuming the phase errors are independent. Of course they aren't, and if you integrate assuming the correct third harmonic mean frequency (10 kHz) when you demodulate the phase errors ought to cancel out and the signal improvement should be better than assuming independent phase errors. But if the instantaneous frequency of the IF is varying a little bit due to the fractional divider, say 1% or so, that does contribute some phase error to the demodulated signal.

I tried to deal with this problem by actually clocking the ADC using a zero-crossing detector. However, there are probably better ways, or maybe the integration time is long enough in practice that this error source is negligible. But it is not true that the period of the IF is constant but it does average out over a long interval.