The OP's battery is four 1.2V rechargeables in series. He calls it 4.8V, but that's just the nominal voltage. If you Google for the maximum fully-charged voltage of a NIMH four-pack, you'll see a wide range of answers, from 5.6V to over 6V.
So he wants to always switch to USB power when it's present, but that may be switching to the *lower* voltage source if the batteries are fully charged, and diodes, even ideal ones, don't want to do that. Even with a diode in the battery line, the mosfet gate (which is at USB voltage) may be enough below the source voltage (the batteries in his preferred mosfet orientation) to turn on the mosfet when it shouldn't be on.
One solution might be to pick a mosfet with a higher threshold voltage - something like 1.5V or 2V. With that, it might be possible to eliminate the battery line diode and replace it with a second mosfet, back to back with the first one, but with the opposite orientation, so you would have opposing body diodes which are bypassed when the mosfets are on. But then you'd have twice the RDSon.
Or maybe @pcprogrammer's TI chip is the better way to go. It's designed to deal with this situation.
Edit: The IRLML9301TRPbF has a 1.3 - 2.4V threshold voltage. A dual P-channel would be the IRF9362PbF.