I think it will actually work, with acceptable brightness range, if you multiplex by segment two segments at a time. By that I mean shift out the values of the A and B segments for all five digits in one 40-clock shift. The values for the other 5 (or 6) segments will be zero. Then when you turn on the transistor, each 164 will output up to 25mA on its A output, and up to 25mA on its B output, with the total output for the chip being 50mA. These are rough values, but they don't do violence to the 164's absolute maximums, particularly if you can control the value of Vcc. The transistor needs to be able to sink up to 250mA, which should be no problem.
Then you would shift out the values for segments C and D, and so forth. So each segment would be lit a bit less than 25% of the time - a little less because of the time required to do the shifting. But if you could leave the segments on for, say, 5ms per pair, the shifting time burden wouldn't be all that great as a percentage - the shifts can be really fast.
This would reduce the amount of shifting you'd have to do. And my guess is that the brightness would be ok - nothing too noticeably dimmer than anything else, no matter how many segments are lit up.
Once you have the values of the individual five digits, I think the rest of the system could be implemented with discrete chips. You could shift out the value of the digit to a 164, and shift out the AND pattern for the two segments you are driving into another 164. Then run the former through a CD4056 7-segment decoder, then AND that output with the second 164 output selecting the segments, and latch that output into an HC165 parallel load shift register, which shifts out the final product. Having done five bytes, you would change the AND value to pick new segments, and do the same value bytes again. After the four AND patterns have been completed, you start over. But of course, you would actually do all this with a microcontroller.