I have a theory that one reason we don't see many your-SAML-implementation-is-completely-broken reports is precisely because it's a gated enterprise feature, so few independent security researchers have the access or ability to poke and prod at them outside of private penetration tests.
The worst bugs here are indeed mostly private, but that's because they're feature bugs inside of people's random products; they're like every other bug in that regard. But people do find and report bugs in the SP libraries.
I agree that SAML is risky to implement; since we agree that Github SAML is an unalloyed good thing, we'd be searching for reasons to disagree at this point.
You take some open source pieces you can (saml, xml, oidc, ssl, jwt) but permissions, groups, user attributes, keys are always per company then the whole thing together has to be supported into end-user applications running on language and frameworks of the day with their own restrictions, so custom.
One can find an open source library to handle part of the SAML or XML in Java, but it doesn't take the right settings or import user attributes as needed or handle URL redirections properly. So the company has to write a ton of authentication code to make it work. It may start from an open-source library but the result is either separate code on top or an outright fork.
The company will end up writing a ton of authentication and authorization code --- it'll do that no matter what, because the application will have its own security logic, like all applications do.
(OIDC doesn't use XML. But the story is the same, with different endpoints.)