You should read up on the terms "bridged" and "differential". I used the latter term, but in the context of speaker amps its the same thing as bridged (albeit probably less specific).
What has been said a few times here is - each connection to the speaker is driven by an amplifier, neither of these connections are at ground. You are still thinking in terms of "single ended" amplifiers, where one side of the speaker is connected to ground, whilst the other side is driven by an amplifier. In that case, your VU meter would have no problems, but your amplifier board - and many modern amplifiers - drive speakers differentially.
So, if you use a scope and measure each speaker terminal with reference to ground (the DC jack) you will see a signal. One will be the opposite of the other - as one side of the speaker is driven high (with respect to ground) the other side will be driven low. So if an amplifier is powered by 12V, it can create ~12V across the speaker in one direction,
but also in the other direction. Like a H-Bridge! Check out the datasheet on page 6:
http://www.st.com/resource/en/datasheet/cd00205863.pdf So what this means is... you have two signals for each speaker, neither of which are ground. You can either just look at one of these signals, which means your VU meter will see a signal sitting at half the power supply voltage, and only give half the indicated voltage (you can use a series capacitor to block DC so it only shows the signal and not the DC offset) ... OR... combine the two differential signals into one, which is why I mentioned a differential amplifier. This creates a single signal, referenced to ground, that will work with your VU meter. It doesn't have to be accurate, so even an LM358 would do.
I'll post an edited picture of your boar din a minute to confirm the ground connection.