The feedback loop keeps the V- input at the V+ (bias) input. This is then anti-modulated (if you will) by the signal!
So the steady state output is Vbias. ... With a split supply it's the same thing: V+ goes to ground and the steady state output is at ground level. Here, "ground" is Vbias.
Yep! exactly so.
Or, if you did add Vbias to the signal (after the cap of course), then the steady state output would be 0V and you do not have to AC couple the output. But you might be adding an RC filter to the input? I guess it depends on how the Vbias is generated and "injected".
You'd need a negative supply to the opamp for it to output 0V (and below), so you wouldn't bother with the extra bias and would just use 0 as V+ at that point. There are some other issues with that suggestion, too... I suggest getting something like LTSpice and experimenting with the circuit there. You do have to be careful trusting simulator results too much, but I think it can answer your questions about this circuit.
AC coupling the output with a capacitor doesn't have the negative supply issue, but you are adding another RC block to consider when thinking about the frequency response. The R and C values matter even in the original circuit, for that matter.
Another option is to buffer VBias with a unity gain opamp and then use it as "ground" in other parts of the circuit (or your headphone outputs, whatever.)