Pretty bad idea to use a 4066 to pass through all 4 lines. Can imagine that it will only work in a handful of cases. Also, you're only going to be able to 'float' the unconnected channel a small amount (say a 4066's input protection diodes worth).
If your aim is to switch USB data and power I can't see why you wouldn't have a common ground. You'd also be better off with switching the USB +5V with perhaps a pair of DMG2305 P channel MOSFETS or frankly any P channel MOSFET - the 4066 has an on-resistance at 5V of perhaps 100-200ohms.
Also, as another poster has pointed out, the BW of 4000 series isn't very high. Probably better with something like a 74HC4066 or better still a 74HC4053 (triple changeover switch). You could build the whole thing with 2/3 resistors, a pair of MOSFET's and a single 74HC4053.