Ok, so it seems to be a two part issue, one is that the analog inputs on that one board I have seem a little screwed up, but secondly for some reason, the arduino script for volume control ignores the last 1/4, I think because they have it map the 1023 values to only 101 positions, I'm going to try increasing that to see if that does it.
But yeah, trying on another board, no issue at all with it going into the negative.