The signal leak through problem is because you aren't grounding the output when the 4066 pass switch is off. Use another switch in U2 to do that, with an inverted control signal.
Yes, you can make it a lot simpler, reducing it to a single chip :- the remaining two switches in U2 can be used with pullup resistors to implement two inverters, then you can use the inverters to implement a latching toggle flipflop with Q and /Q outputs to drive the signal switches.

I suggest 4.7K pullups on each inverter output, a 47K feedback resistor and a 470K charging resistor for the toggle capacitor. To get it to power up in a known state, add a 1nF cap from the input of the left inverter to ground.
However it would be better and easier to use a CD4053 triple SPDT switch. The sections used as inverters would no longer need pullup resistors and the NC pin of the signal switch could be used to ground the output (COM pin) when it isn't connected to the signal source on the NO pin.