I think what happens is it overshoots, then once the output decays below the true peak, it again overshoots...
Given a LF sine wave input the response looks good "at a distance", however this peak "restores" with a periodic (1 or 2 seconds) classic RC decay and the "restore spike" is about 300mV amplitude.
This seems (in retrospect) what would be expected. I never really thought about it until the comparator I was driving starting flashing its indicator LED at rougly 1Hz.
I can't afford sufficient comparator hysteresis to swamp this effect. It would be better if I could have a smaller and faster overshoot - would feed-forward be worth trying? A smaller capacitor, larger resistor would help somewhat, but not enough.
There are many solutions involving different Peak detect principles and/or more elaborate reset methods, but I was wanting a very simple design, which I why I chose this one.
Which op-amp are you using? Are you sure it's going to be happy driving a 10uF cap? 10uF seems pretty excessive for a peak detector.
I would raise the 'Z' ground vokage node a little using a voltage divider such that the divided voltage is a little higher
than the Opamp input offset voltage plus the maximum input noise voltage.
Just experiment a little; I would start with a divider of 10K to +5V and 22R to GND, which yields 11 mV offset.