So I think I may have found the best combination for this transformer. Connecting the brown + white/black wire (the one that seems to produce the most current) and used the yellow pair (8.8VAC) and that actually bumped it up to a solid 14V at idle and with a car stereo playing a single speaker at full volume, it only dropped to about 12V or so. This is more what I was expecting from this transformer
But... It's still not enough. I'm starting to realize that even though the transformer is physcially large, it has 4 different outputs and at least 3 inputs. There's a lot going on and it doesn't seem to carry nearly as much current as I original thought.
So in that case, I think I'm either going to find another one that fits my application, build my own, or custom order one (probably from Agile Magnetics). Depends on the cost. It's worth calling for at least a quote though. Unless anybody knows of a transformer that'd be able to provide about 5A at 12-24 volts. I don't mind if it's over 12 because I can just regulate it from there. This way I can make it variable.