You didn't give details on your charging source or battery, but with lipo batteries it's necessary to isolate the charging process from the load current. That's because the charging IC monitors the battery voltage and current to determine when to shut off charging. So when the charger is connected, it separately provides charging current to the charger IC and battery, but also directly provides current to the load. The load current doesn't flow through the charger. Here's a discussion about how this is done:
http://blog.zakkemble.net/a-lithium-battery-charger-with-load-sharing/Look for the schematic labeled "Charge Circuit with Load Sharing", as well as his explanation.
Basically, you need three additional parts - a P-channel mosfet, a shottky diode, and a resistor. When the USB charging source is not plugged in, the mosfet gate is pulled low by the resistor, which turns it on and allows the battery to power the load. The voltage drop is very small. When charging power is applied, the mosfet's gate goes high, which turns it off, the load is powered through the diode, and the battery is isolated from the high USB voltage by the mosfet's body diode. With this setup, battery charging can proceed to completion and shut down while the charging source continues to power the load.
Downstream from all of this would be a voltage regulator of some kind, and then the load itself.