LED stops flashing after removing programmer

[Kremmen]

Yup, uses debugWire as one of the optional protocols. That in turn communicates over the reset pin so you do not want to leave the debugger/programmer pulling reset down when powered off.
debugWire is used for debugging and ISP for flash programming; both connections are made using the 6 pin header so you would want to disconnect the prog cable when starting normally with the programmer off.