Yes by sagging I meant the drop in battery voltage under load.
Your readings will only be as accurate as your ADC reference. If the reference (Vcc) is changing as you switch the load you may get strange readings. Monitor the Vcc rail with an oscilloscope to make sure it's nice and stable. As you are not using the battery for your Vcc this is probably not the problem but worth checking.
Also disregard the my other point about moving the load. 5v will switch it nicely.
Checked the Vcc and it is pretty stable.
I think I know the problem now: bad connection with the battery + resistance of the wires. Solved the problem by using a neodymium magnet to ensure a strong connection and used a software calibration to compensate. Now the error is low enough.
The software calibration used is a linear equation, and i got the parameters by subtracting the value measured by the ADC from the value from my multimeter, then plot the Voltage(Vadc - Vmultimeter) vs current graph, and traced the linear trend.
Thank you tomlut for all the help and tips