Could you just put a high value resistor from the gate to the source of the mosfet to drain that leaking current away from the gate?
Yes, but it's a P-channel mosfet, so the resistor is to ground so the mosfet will turn on when the Schottky path is turned off. Leakage would raise the gate voltage, which could prevent the mosfet from turning on fully. But yes, the ultimate question is the value of that resistor to make sure leakage doesn't mess things up. I mean, I could just make it 1K, but I hope something a little less wasteful will work.
I'll attach the circuit which currently calls for a regular rectifier diode. For a variety of reasons, it needs to be a Schottky. Leakage may actually not be a problem at all. If the mosfet doesn't turn on, then there isn't going to be much voltage on the cathode to leak through in the first place. Anyway, I just need the regulator to switch over to battery power from USB power with no interruption, so the Arduino being powered won't reset. It's supposed to act like a UPS.