"Demux" (in Tektronix terminology, not sure if this is standard) is the part between the ADC and the memory. I think this is now integrated into a single ASIC (finally...), but conceptually, it divides the "huge" ADC data rate (100 GByte/s in this case) into many different channels, so each channel can be handled with off-the-shelf memory. (See
https://debugmo.de/2013/03/whats-inside-tektronix-dpo5034/ for example - I apologize for _still_ not having fixed the image links, use
https://goo.gl/photos/dGy6WveTGwSdKgC4A directly if you want)
In the MSO/DPO5xxx (and DPO4xxx, and MSO4xxx and MDO4xxx), for example there are 2 ADC chips, each handling 10GS/s (from 1-2 channels); each ADC is connected to two Demux, each Demux has 4 memory chips. This means the data rate (20GByte/s in this case) is distributed to 16 memory channels; at 1.25GByte/s, running them at 800 MT/s @ 16-bit (DDR800) is sufficient to store the full rate. (The Demux does more - it can apply FIR filters, read back data etc.)
For the MSO5 and MSO6, Tektronix revamped their architecture and now has an integrated ASIC with ADC , Demux and memory controllers. I don't have more details than that, but overall it's still the same concept, just more tightly integrated.
So the sample rate bottleneck is not just the ADC, but also the memory bandwidth. More memory bandwidth would require more memory channels, and they are costly (mostly in terms of board space and ASIC pins).
With the ability to "bypass" the ADC, and directly feed the Logic signals to the Demux (slightly simplified - of course there are level shifter/comperators etc. involved), you can use the available bandwidth for either analog channels or digital channels. Each analog channel will replace 8 digital channels (which is the ADC depth at full speed). In either combination, the Demux are running at full rate. It's not matter of "breaking out" more pins. It's a matter of running into the primary (or secondary, if you count the ADC as primary) physical limitation of the device.
(Now of course we could be talking about running 4 Analog channels at half speed together with all digital channels at half speed, but - thankfully? - Tektronix didn't chose to go down that route.)
The MSO5xxx (not MSO5) MSO limitations are because the MSO is handled not by the main acquisition path (Demux -> DDR memory), but by a seperate path (Trigger ASIC -> plain old SDRAM), which is limited at 500MS/s (and has some other arcane limits).