That's nasty if you need to go past 115Kbaud. You need a bandwidth at least an order of magnitude better than the baud rate to keep the rise and fall time of the edges acceptable, so to support 3Mbaud, you need switches that will pass 30MHz.
You'll probably get away with a 74HC4066 for each device, but the 220R worst case on resistance may be an issue if any of your devices have low impedance inputs or if you need to use higher baud rates, so you may prefer a digital only solution using non-inverting quad tristate buffers, e.g 74HCT136 (which would be OK for 3Mbaud operation). Tristate the outputs of the buffers, or open the switches for all the powered down devices. Only the single currently powered device has its switches on or its buffers enabled.
You *MUST* *NOT* activate the switches/buffers for more than one device at a time or you will risk causing a level conflict (bus contention) that will result in malfunction and may cause damage.