I think the main reason the requirement to publish the sources is limited to the person who receives the binary is simply practical. For one, there is basically no way to sue as a third party to a contract - even if the contract required you to publish all of your code openly, someone who didn't receive the binary can't really have standing to sue if you just don't publish it. Also, at the time the GPL was created, sending source code carried some measurable cost, so making it public would have been at least mildly expensive.