The LM324 is an early op-amp, and it could do funny things when you are getting close to the rail voltages.
Do study the datasheet!
To verify if that is the reason, do add some negative supply to the op-amp(s), and redesign your circuit with a more modern rail-to-rail op-amp, if proven so.
PS: Do include the schematics!
I (and probably the others) do not keep that at my fingertips!
Regards, Peter