A few things. First, the TL3116 is 0 to 2.5V common mode input so you should restrict the voltage on the capacitor to a range of 0 to 2.5V. You may also be allowing the current mirror transistor to saturate, and it cannot get out of saturation until the base storage charge is dissipate, Restricting the voltage on the 300p cap means you can stop the current source transistor from saturating, and it will behave far more perfectly.
Thirdly, reducing the maximum voltage on the 330pf cap means you can add emitter resistors in the current mirror, and it will behave far more accurately and predictably.
I am not sure why you have added a 1n4148 to the BS170 drain. It will mean the reset voltage on the 300p cap is not constant, and with the significant drain capacitance of the BS170, is will add some non-linearity to the charging curve.
So I would eliminate the 1N4148. I would increase the 330p cap to 1nF to triple the charging current to 1mA for a 0 t0 2V in 2usecs. Add 680 ohm resistors to the emitter legs of the current mirror transistors. Add another pnp transistor with its emitter to the top of the 1nF (was 300p) cap, collector to 0v, and base to about 1.6V set by a resistive divider (2k7 and 5k6 possibly). That will prevent the current mirror transistors from ever saturating.
I do not know the external input voltage range to the TL3116, but if it goes over 2V, you may need to put a voltage divider on the input. It has a low current input, so that would probably be fine.
The emitter resistors couple be increased up to about 1k5 (with the 1nF capacitor) and the bigger the resistor, the more predictable the current mirror behavior. With the emitter resistor, the 2uS ramp may be consistent enough to allow you to replace Rv1 with a fixed resistor (perhaps 3k3 for the 680 ohm emitter resistors).
It should now work fine.
Richard.