EDIT: and would you then also review every commit to make sure nothing bad gets introduced? No, at some point you have to place trust in the vendor, the developers, independent audits, etc.
Assuming all three match, you know that the binary matches the source.
Someone who is more technically inclined can probably go into more detail on this.
For the curious: we actually were intentional about finding these, by compiling many programs with the same parameters on different machines. One with a 32 bit OS and toolchain, the other one on a 64 bit machine, and we would get alerted when we produced binaries with a different checksum.