EDIT: and would you then also review every commit to make sure nothing bad gets introduced? No, at some point you have to place trust in the vendor, the developers, independent audits, etc.
Assuming all three match, you know that the binary matches the source.
Someone who is more technically inclined can probably go into more detail on this.