Now instead of one layer with hardware contact, you have two (assuming you want performance too). Twice the attack surface.
That is correct. This is probably why privesc exploits are much more expensive than adobe reader exploits.
You are kind of arguing against yourself here.