I don't see an NTC working very well.
Few ideas:
- use a single 12V battery and run a 24V boost converter module off of it, with an enable input. Enable the supply only after connecting it.
- put diode on output of internal power supply, then run the battery to that, so the internal capacitor will not charge up
- add in an active inrush limiting circuit (mosfet, or dedicated IC)