Sure, it seems simple enough but it's not clear to me which direction the device is controlling. I guess the question is: Who is the master and who is the slave?
Have you considered a need for control signals on the A1..4 ports? Data Available, Transaction Complete, that kind of thing. Are these ports used for different protocols simultaneously?
A sort of Truth Table describing what the Addr1..4 signals are supposed to do would help. Remember, I2C, in particular, is a master/slave arrangement that allows for multiple masters. If something comes in on I2C how is the address field and R/W' signal going to be handled.
I think it's going to be messy because you don't have control signals. Maybe there is some room in the Addr1..4 signals to select between data from/to the port and control to/from the port.
Electrically, this is no big deal but you need to spend a lot more time detailing what you expect to happen.