I think you have stepped into typical problem of this type of peak detector.
In the positive half cycle the OP amp works as expected, but in the negative half cycle, the OP amp actually goes into negative saturation making the diode reverse biased. This negative saturation is problem if you do not select the right op amp. Typically, when a OP amp goes into saturation (negative or positive), it takes a much much longer time for the OP amp to recover back to normal operation. Some OP amp just does not recover.
If you try to reduce the frequency to say 10Khz, you might get the peak detector working with MCP6004. Unfortunately, this is not always mentioned in the OP specification. Try to use a another peak detector design. This is one of the problem with this simple peak detector. Typically for these type of design, you need a high speed OP (a few order faster than the highest freq) to get this to work, so that the recovery time is still within the response time of you max operating frequency.
If you look at the bandwidth for MCP6024, it is 10Mhz, but for MCP6002, its only 1Mhz. You might think that 1Mhz should be able to handle 100Khz signal, but no, the recovery is much longer than the response time of the system. When that happen, the the capacitor is charged but never discharge resulting in you observed when the IC is changed.