It's important to note that GrapheneOS is not some niche barely-used project. It has existed since 2014 and is used by multiple hundreds of thousands of people at this point. There are also many eyes on the project through people forking it to make their own products, people maintaining their own builds etc. GrapheneOS is also reproducible in addition being open source.
On our side, we are very particular about accepting outside contributions if they don't need meet our standards, and code is heavily reviewed within our team before being merged.
I'd also recommend giving https://grapheneos.org/faq#audit a read through.
All in all, your concern, while valid, isn't something that's likely to happen precisely because we're very aware of situations where it has (see xz) and are therefore very vigilant. The kind of thing you're worried about isn't likely to come from a big project like GrapheneOS that has many eyes on it, but rather something small that's used everywhere and barely has a couple of devs working on it, if that (again, see xz).