That protects you from hypervisor and hardware attacks. The only thing you need to trust is that none of the guests can induce the windowing system to incorrectly direct output, and that the windowing system can enforce access control (mandatory or discretionary) to the various guests.
The basic/extant version of this is putting a bunch of discrete devices in different security domains on serial ports, and then having a trusted console server to intermediate everything. A console server is vastly simpler than a full graphic windowing system like X Windows, much easier to audit, and more secure.