True, neither the absence of an identified vuln in published source code, nor the absence of published source code can guarantee that you don't have vulns. And sure, a bad-faith operator can always back-door the server and run different code.
But, a good-faith operator can find and fix bugs faster if they operate in the open and in collaboration with the community. "Given enough eyeballs, all bugs are shallow" etc.