The only way to get decent signal integrity (SI) with that thing is to go 50 ohm: terminate the signals from the mux in 50 ohm at your scope, using (in order of decreasing SI) the internal termination if present, a feed-through 50 ohm terminator, or a BNC tee with 50 ohm terminator. You will also need to feed the mux from a 50 ohm source, for example a 50 ohm signal generator, an active probe designed for 50 ohm inputs, or a restive divider probe as for example described by Howard Johnson or Doug Smith. Neither of these options are very nice for general-purpose low-frequency probing (eg. power supplies, audio).