^ I concur, since you have a diode in there already...
A boost converter wouldn't waste much energy when the USB takes over, anyway.
But for good measure, sure why not?
Just in case you didn't realize it... if you want to be able to digitally turn the device on/off when the USB is not plugged in, for instance, you can't use the EN pin for the reason stated in my previous post. If you need to do that, put the FET between the battery and the boost circuit. And switch it on/off from there. You will still need the rectifier diode, though.