This sounds like many incorrect design decisions being made...
- LM324 has large offset (max 3mV --> an output of 9V from your circuit), offset requirement unstated
- LM324 has limited gain (min 25 V/mV, you're asking 3 V/mV), linearity requirement unstated
- LM324 has high noise (~23nV/rtHz?), noise requirement unstated
- LM324 has low bandwidth (GBW ~ 1MHz, a roll-off of 300Hz in your circuit), bandwidth requirement unstated
- The SS495 has an amplifier internal, why do you need more gain?
- Even the SS495A1 (best grade?) has 25 times more offset than the LM324 does, how will you trim or zero-offset this?
- This is for a gokart, what function is it serving? If electrical current sense, you will be far better served with a proper DC current sensor. If motor or engine sensing, a stronger magnet, or closer proximity, or logic type Halle effect sensor, or an inductive or optical sensor, should be used instead.
As for your observation, likely both components -- the sensor and the LM324 -- exhibit strain related drift. This occurs because silicon is sensitive to strain, changing the bandgap and therefore all voltage drops proportional to it (like the base-emitter voltage in the input differential amplifier). (Not exactly, but close enough.) The IC is built to minimize this type of error (and due to thermal influence as well), but amplified thousands of times, it doesn't quite cancel out and you observe the output changing rapidly with strain.
This can be fixed by using a self-zeroing type amplifier, like the OPA2180, AD8628, or simply a better one, like OP07, OP37 or LT1012. But it'll be much easier to deal with the original problem, I think.
Tim