The general solution is to use a r2r opamp - it doesn't solve the problem but just alleviates it to a point, hopefully, that it doesn't matter as much to you.
You did good in this case in that you picked LM324 - with its pnp stage, it could go to a fairly low level before failing. Had you used an opamp with a NPN (or worse yet, jfet) input stage, the failure would have shown much earlier.
A side note: a r2r (input) opamp is really two opamps in one, one with pnp input (going to the negative rail) and another with npn input stage (going to the positive rail).
But you should not expect to much, as long as you are in a single rail setup - 10mv to the rails is fairly decent performance, even for r2r opamps.