Maybe a bit of an overkill, but if one would use an RPi ZeroW, you could simply stack multiple of them (together with a video-grabber) in one case, each providing KVM for one host. If you only want to use one network port, add an internal network switch to the package, have one the RPis doing the frontend to it (requires a second network port on it).