The thing that could bite is the latency, so do carefully consider what kind of a protocol you'll use.
Personally, I'd use a combined substream-and-length byte, with 8-N high bits the substream channel, and N low bits indicating length; maybe 4:4, with zero channel reserved for transport and configuration messages, and zero byte (zero channel, zero length) reserved for synchronization. That way, sending 17 zero bytes ensures the other end is synchronized, no matter what was sent before.
For example, if one end is an SBC or other built in hardware port, and the other end a microcontroller with suitable serial ports, then zero channel could be used to configure each external port. In fact, one could then even write a simple Linux kernel driver that when bound to a serial port, provides the end point serial ports as device nodes, uses the zero channel for port configuration, and multiplexes the communications on the actual hardware connection.
(Alternatively, you could use say 1 bit to indicate control message (even support RTS/CTS handshaking using control messages), 3 bits the channel, and 4 bits for the message length. You do want to keep the message length small, and implement some kind of (round-robin?) scheduling between the channels, to keep the latencies on each channel acceptable.)
This is very similar to what I've been playing with on my Odroid HC-1: it doesn't expose any GPIOs, only a basic UART (VCC, RX, TX, GND) with 1.8V logic levels. One of my projects is to use a Teensy 4.x (with 74LVC1T45 level shifters) to interface to the UART, multiplexing the single connection between serial console (exposed by the Teensy via Ethernet and/or USB) and a 2.8" IPS panel 320x240 display. There, too, the main issue is the multiplexing protocol, except that serial messages tend to be short, whereas display commands and data tend to be long, which complicates things if one wants to be efficient. (The Odroid HC-1 does have USB ports, so I could use that instead, however. It's fun to play with...)