Don't give up just yet, you're nearly there!
Having a good look at the data sheet, again I must re-iterate that an electrolytic cap is needed on the power pin to ground, data sheet recommends 10uF or higher, I would actually use also a 100uF to 470uF to keep the rail stable especially if it's not supplied by a low impedance source like a battery.
I don't see this rail cap on your latest schematic.
In the evaluation module, they strongly recommend a minimum output capacitor of 33uF for headphones and or line output, It may need this for stability? your circuit only shows 1uF output caps.
See
http://www.ti.com/lit/ug/slou084/slou084.pdfAlso, the data sheet states that the amplifier is unstable if the source impedance around the - input is more than 50K and recommends a small cap across the feedback resistor in such cases, I'm wondering if it's worth just adding a small cap to prevent HF oscillation, see page 5 of the data sheet.
The LM324 is not a very good op amp, consider a CMOS rail to rail input and output op amp to maximise your dynamic range especially since you only have 3.3v to play with.
Just curious how you're sending audio to the detector, are you using some sort of modulation?