A lot of these things seem to "help" by inserting their own USB host device that acts as a proxy for attached USB peripherals. This allows them to do (ostensibly) useful useful things such as intercepting keypresses and responding to them (e.g. by switching to another input).
Don't get one of those, get a physical switch that just connects the leads to the correct port when pressed. I got this USB 2.0 switch (all you need for input devices) for $12 [0]. I'm sure there are others available with more ports, but this is all I needed.
Though, I also use a KVM to switch over the video but it's separate.