That's what I'm thinking, or worst case, one that communicated via USB to TTL adaptors. Or an ESP-based board that connects over WiFi. Or BT-enabled board.
Combined with a HDMI switch, you could then control multiple devices with just one Pi+Cap device.
I've been looking to do something similar for a while, albeit possibly with Windows as that's where I've (very rusty Delphi) programming experience.