I agree with abyrvalg that the FPGA strap options is most likely setting hard limits; so MSO=(6000/7000), /MSO=(5000). Probably these decisions were wired in before the marketing department started gearing up.
Obviously the question is, did Agilent purposely cripple the scopes to prevent 2-channel serial decodes, or do they disallow it to prevent an invalid condition.
For example, I could imagine that the implementation is such that the MUXes in the FPGA (or ASIC) want to talk to all digital + All 4 analog channels, but if only 2 channels are active, some handshaking never happens and the decoder hangs, but if all four channels are activated on a 2-channel scope, then some handshaking at the A/D FIFO doesn't happen, and the scope hangs, etc.
So, it might be interesting to follow the serial decode code, both option setup, and decode setup to see how it sets up the hardware. I think that if it's not a hardware limitation, that is where the answer lies. BTW, do the 5000 series have the serial trigger option, or nothing at all?
Dave