I am definitely not an expert, but:
A purely mechanical connector might not have clean enough signal paths. Also, that connector does not say whether it is make-before-break or break-before-make; you'll want it to be break-before-make (non-shorting) so that when changing, the two USB cables D+ and D- won't ever be connected together.
Personally, I'd try something like
CRFS-2202W instead (and that assumes you have diodes or P-MOSFETs in diode configuration to protect the two USB +5V lines from being backfed from the other connector). Or, better yet, perhaps a Switchcraft C63212L (either with +5V commoned, or if the device needs less than 300mA, then with +5V and GNDs).
That 4PDT Switchcraft C63212L, assuming the device needs less than 300 mA, would probably soothe my paranoid mind the most, switching all four USB lines (+5V, D+, D-, GND). That way, if I accidentally connected the two cables to two different computers in different ground potentials, no harm no foul; the switch would completely isolate the unused connector.
Again, I'm just a bumblef*ck, and have no idea if any of those switches actually work.