I thought the PCI passthrough security model assumes that the guest does compromise the GPU, and the guest-controlled GPU is isolated from the rest of the system using the IOMMU.
Do you mean these controls are porous by design or are you talking about bugs in the IOMMU protections?
>>zurn+(OP)
First of all guest can easily update firmware on that device. As stated before GPUs are really complex devices and some part of them might be badly documented, like there is whole HDCP / DRM support that isn't documented at all. Of something like that happen you'll never find out.
Next time your PC going to boot your GPU will be initialized with host BIOS / UEFI long before kernel get possibility to limit it with IOMMU.
>>SXX+A7
Interesting point. I hope this threat is something GPU vendors address, since secure virtualized GPU access has been a marketed feature for a while (at least from AMD). Quick googling drew a blank sadly.