Man, this is getting weird.
First of all, yes the caps are new. Tried different ones - same result.
And I did check very carefully, to make sure the mains leads arent having a continuity to anything.
Anyways, I just put a couple of 470R 1W resistors from each output to ground.
Now the 7912 shows a nice -11.8 Happy happy.
However, now the 7812 only shows 0.7
The resistor across the 7912 does get a bit warm, whereas the one across the 7812 stays cool.
EDIT:
Just changed the 7812 again, and it turns out that was a bad one. Now the 7912 is around 11,8 and the 7812 is around 11,6 under load.
However unloaded they are right back at -17,8 and +11,7