I'm not a tinfoil hat, but security can't hang it's hat on the kindness of strangers.
Given that SSO is a massive security win and has been a game changer for removing passwords, I think it's been shown that delegation is extremely effective.
1. Instead of needing 100 passwords, which increases the chance of users just choosing something and repeating it, you have 1 password.
2. Similarly, instead of needing 2FA on 100 sites they can just have 2FA on their SSO. In fact, the other sites don't even need to support 2FA - you get that "for free" with SSO.
3. SSO providers implement auth really well. They make it smooth, as in "I don't have to reauth when it's obviously me" and safe, as in "that might not be a valid auth, let's get them to 2fa again".
Of course, if you have a password manager then (1) is not a problem. But SSO is a lot simpler for users.
a) SSO has no financial cost. Hardware keys do.
b) SSO has been implemented and standard for years and is trivial for sites to support, hardware keys are much newer and are still rarely supported for authentication.
c) You can use hardware keys with SSO, which I'd recommend, and now you've gotten the benefits of both.