A switch mode regulator draws a constant input power while regulating the output:
If your input voltage is low, it needs a high current to deliver the necessary power. If your power source has a high impedance or a current limit, it results in this behaviour you see, because the switching regulator draws so much current when starting at low input voltages.
One possible workaround is implementing some undervoltage shutdown that disables the switching regulator when the input voltage is too low.