If nothign else, the slower slaves will need to decode the address phase correctly in order to know they're NOT being accessed. If they see a blur, you can't be sure they won't think they recognise their own address every so often, and start ACKing when you don't want them to.
So, it's another vote here for separating the buses. I suggest an analogue switch IC, or perhaps more easily, just use a couple of GPIO pins on your processor to drive the slower devices on one bus, and keep the display on its own on a second bus.