Remote attestation is the true enemy of your freedom. The power of the authoritarian corporatocracy to force you to use only the (entire) systems they control. It's worth reading https://www.gnu.org/philosophy/right-to-read.en.html again just to see how prescient Stallman was.
I get the issue with Pluton but TPM is only a dedicated and certified secure key and random number generator that does a better job than CPUs doing it in software, and it's also a secure enclave for storing your encryption keys. Would you rather store the keys in memory where they can be easily grabbed by malicious apps like Mimikatz? Macs had the same feature for years in the T2 chip.
It's the exact system that enables wireless payment and other strong security features on your phone.
So having TPM on PCs and using it for its interested purpose is a boon for everyone's security so I don't see the issue, just FUD.
So in worst case, if your attestation server is very strict, any new binary installed on your machine will prevent it from booting or satisfying the attestation. This is the main concern that TPM enables.