Very interesting stuff, thank you.
Yes, I think it's likely that the GPS has it's own granularity due to some internal clock frequency.
But as I understand it a TDC won't solve that problem, it's just a way to give a binary representation of the time delay between two pulse with very high resolution. I need to actually generate my own 100MHz reference clock and then synchronize this with the 1PPS "on average" to even out that sawtooth pattern. I guess the Nutt method mentioned in one of your links could be used to measure, very accurately, the time between two 1PPS pulses and then this in turn could be the input to a DPLL FPGA block.
I will have to investigate further.