Well, just throw a cap on the Q1 base and see how much it changes.
Your emitter resistor is only having 0.25 V, that's not a lot.
The more voltage on the emitter resistor, the more accurate your current will be.
Normally you'd use a zener supply on the base and a couple of volts from the emitter resistor.
Ok, you don't have the headroom, because you're running 3V signal on a 5V supply.
Since you don't have the headroom, just use a current mirror, the least headroom topology you can do.
Another PNP with the collector/base going to the only resistor to ground.
Q1 emitter to +5, base to new transistor collector/base, collector to cap.
I presume that you actually want to do something with the ramp and not just take the comparator output?
Put a unity follower opamp between the cap and the comparator.
Now you have a low impedance output that you can use.
That will get rid of the 10 microamps that the comparator is taking.