To explain what may be happening, the USB power supply might be tripping out on over current protection. When you initially connect the converter, it'll have an inrush current spike, which may be triggering the PSU protection circuitry.
USB specs is usually 500mA for a USB port, but many devices will supply 1A or 2A, sometimes more on chargers.
what you could try, is a sufficiently rated low ohm resistor in series with the supply, to limit the switch-on spike in current. You can try a 2 ohm or 1 ohm resistor. The downside of this is it will create a voltage drop across the resistor in use.
Or maybe you can use an NTC? I'm not sure that would work though in low voltage circuits. It depends how much current you circuit draws in use, because NTC's have a minimum current draw to work, and might have too much voltage drop at low currents.
I've seem some articles saying you can use a FET for inrush current limiting too, so maybe that is an angle worth investigating.