I'm not going to comment on the crap design as a whole, though I will compliment the OP on their neat assembly work.
However:
Unfortunately if there is no means of fixing the quiescent bias votage, then trying to alter the mid point will change that also. In fact, with just a simple resistive setting of the output stage current, It will also change asymmetrically in sympathy with the signal, causing amplitude nonlinearity.
The op needs to replace the simple pot/diodes whatever, with a transistor Vbe multiplier circuit, as previously suggested, to minimise the change of bias voltage, (output quiescent current bias), as happens when the current in the driver stage changes.
Without either underrunning the bias or having a correctly designed circuit that has thermal feedback this amplifier could eventually die from thermal runaway.
To answer the OP question regarding which is the more important, then that would be the quiescent current bias, the centerpoint voltage being incorrect will only limit the output swing in one or other direction from the quiescent output voltage point.
Just search 'amplified diode circuit', implement it correctly, and life will become much simpler.
Regards,
Xena