Why not use a gigabit Ethernet MAC instead?
You have to agree really, the ping times are a lot faster with GBE. Make sure to use a board that uses real Ethernet and especially real interrupts for the NTP (and not filtered through USB like the RPIs up through 3).
Is the difference significant? Opinions vary a lot on this.