A better way would be a relay and microcontroller, which you could also potentially use for the USB HID stuff too.
In theory, you could have a HDMI switch, a single Pi and HDMI grabber, and just a microcontroller board for each target machine that could have power button control, keyboard emulation etc built in.