First check - do you still get pops & clicks when switching between silent audio sources (driven, not floating)? If so, it's almost certainly charge injection. This is probably from the switching device itself, but it just might be capacitive coupling from the logic control line. One trick I have used in the past is to implement the switching at a virtual earth point, so that the voltages on the switch terminals do not change. You need two switches, one normally open and one normally closed. Arrange these so that the input signal is routed, via a resistor, either to the virtual earth input of an inverting OPAMP stage, or to real earth (the switches go at the 'earthy' end of the input resistor).