MOSFET or other switching circuit to disconnect two separate grounds

[Finndersen]

Ok so I have a project where I'm powering a Rpi with either external DC supply or a USB power battery bank thing.
The USB power bank has a charging input and power output.


It's done in some strange way where the input and output grounds are disconnected from each other when the battery pack is not in operation (either charging or supplying power)
When it is in operation, the grounds be come connected.
However, if the input and output ground are connected together while the pack is meant to be off, it puts it into some strange semi-on state where it has 3.8V at output terminal, causing an LED to come on and the Rpi to be partially powered but not work. Basically non-ideal.

When connected in the shown configuration, the Rpi becomes a bridge connecting the battery pack output ground to input ground, which causes this problem.

The simplest solution is to have a physical switch between external DC supply and battery pack charging input, which is only turned on when external supply is powered in order to charge battery.
The switch could also be turned on when the battery pack is active and supplying power to the GPIO input of the Rpi. There is an internal ideal diode in the Rpi which will prevent the battery pack charging itself.
But the connection from output ground to input ground must be severed when the system is meant to be completely off.

I've played around with some N and P channel MOSFETS but havent  found a configuration which works.
For example with a simple N-channel MOSFET switch, with battery input ground connected to Source, Rpi output ground to Drain, and external DC positive to Gate, the battery input ground must be tied to external supply ground in order for the positive VGS reference to work.

Can anyone help? Having no physical switch would be ideal

Cheers