Yes the ground pin can be removed and you will pull the return current through a VSS clamp diode structure instead.
Ok, but why does it stop when one of the blinking LED connection is removed?
I did some experiments in LTspice to learn more about the details. This is the normal behaviour of a digital output pin, e.g. for one of the LED blinking outputs:
It push/pulls the output pin very nicely to Vcc and ground.
The output MOSFETs have internal body diodes, or manufacturers put in extra diodes for ESD protection. The IRF MOSFETs in this example have internal diodes. You can see the effect when the outputs are turned off, then the diodes provide some kind of ESD protection:
Finally this is the setup when the ground connection is removed:
R2 is a placeholder for the CPU core. The internal ground in the chip is gnd2. When the LED is turned on (V(gate)=low), M1 conducts and the output is connected to Vcc. This means no current can flow in the path Vcc->CPU core R2->body diode of M2->output, because output is (nearly) at the same voltage level as gnd2 (because of the low impedance connection of M1, compared to the LED and R1 connection). But when the LED is turned off (V(gate)=high), M2 conducts and M1 not. This means current can flow in the path Vcc->CPU core R2->gnd2->body diode of M2->output->R1->LED D1->gnd. So if you have two output pins and they are alternating high and low, gnd2 is always at about 0.5V. But as soon as you remove one LED, gnd2 goes up to Vcc and the CPU core stops working. Note that you can connect any input pin to ground and it will still work if one LED is disconnected, because as we've seen before, when both output FETs are disabled for an input pin, current can flow through the body diode of M2 of this pin.
PS: Regarding my last question: You can tell that it is the body diode of the n-channel MOSFET M2 that is conducting and not the source-drain connection, because if you change R2 to 1k, gnd2 increases to 1.9V (because of the higher voltage drop of the diode at higher currents), but output is only at 1.3V, so there is the diode. If voltage is flowing from drain to source, there is only the usual very low voltage drop because of the Rdson resistor.
PPS: you can power the microcontroller with no connection to Vcc or ground at all: Just connect one input pin to Vcc and one input pin to ground
(Vcc has to be sufficient high for the voltage drop of the two ESD protection diodes)
Sorry for the long-winded explanation, this is very basic stuff, but I didn't know it.