I would use a dedicated CH554G per computer, and a SI8621 or ISO7221 digital isolator per CH554G, so that each USB device has their own local ground and VCC (USB 5V).
The keyboard and mouse would be connected to a more powerful microcontroller with USB host capability. I myself would probably use a Teensy 4.0 with an external off-the-shelf USB hub (soldered directly to the USB host pads on the Teensy), because I'm familiar with it, and know it can do this – and isn't too expensive ($20); and it has five UARTs on the easily accessible pins (7 if you use the pads on the bottom as well) to connect to each CH554G. Both
Teensy and the
CH554Gs can be programmed in the Arduino environment.
USB HID is limited to 64000 bytes/sec per device, so using an UART (serial connection) between Teensy and each CH554G would be perfectly okay. You even have ample SPI and I2C buses left over, in case you wanted to add a display for extra control of the unit; I like the small OLED displays, as well as
BuyDisplay (AKA EastRising) IPS TFT displays (using ILI9431, ST7735, or ST7789 controllers), as they look nice and there are existing Arduino libraries supporting these. (Having a small flip-up 128x32-pixel 0.91" OLED display per USB port might look nice, although I don't know if it would be useful per se...)
At minimum, I would at least add a button and two leds per computer: one led showing whether it is connected, and another to show whether it is actively forwarding the keypresses/mouse events, and a button to select between the two. A 2.8" display with capacitive touch would make it simpler (as you could show the connected devices graphically, using icons as the buttons selecting operation), but costs a bit (about USD $20 from BuyDisplay; they also sell these on
eBay).
For even niftier use cases, I'd actually switch to a Teensy 4.1 and get the Ethernet kit too, so that the same device could be used as a network-controlled keyboard-mouse combo, and an SD card could be used to hold full-display icons in uncompressed form (320x240), say created in Gimp and saved in PPM form then processed through a simple program (PPM itself is uncompressed; trivial to load and save) to convert to the desired form.
Note that the digital isolators are not optional. When you have multiple machines, you really don't have any guarantees their USB grounds are at the exact same potential. And if they are not, and you connect them, some current will flow: you have a ground loop. You want to avoid this, because in my experience, it does odd stuff to hardware. (Or maybe I'm just unlucky, or extra butter-fingery, having the Magic Smoke escape from my devices when I do things like that.) The digital isolators not only act as a barrier, but they also mean each slave CH554G module is essentially identical. This kind of modularity makes things easier. On the Teensy end, only the Serial object changes depending on the slave.
If you don't like the CH554G's, any microcontroller with native USB interface will do, for example ATmega32u4 (which are used in many custom keyboard projects).