Just saw the schematic you added. See the little arrow in the MOSFET symbol? That's a diode, formed by the FET substrate.
If your signal happened to be under 0.6V peak, the diode would never conduct, and the MOSFET would appear to be symmetric.
Just food for thought
If you got a MOSFET with a separate pin for the substrate (not worth it - IIRC only one is left on the market, probably soon to be EOL'd, kind of expensive, and with crap specs), you could bias that pin negative and then this circuit would work.
Another problem, given the schematic you just added: You're not switching a
signal, you're switching
power, at least enough to drive headphones. You'll need large enough capacitors to carry that current, and a switch with a low enough ON-resistance as well.