I doubt that many people rebuild the app at each update to check that the new binaries match the ones provided by their store. If, for example, the PlayStore distributed at large a binary that doesn't match the published sources, some dedicated user would probably spot the issue.
However, the PlayStore (and Signal, but it's not even necessary for the following) being under US jurisdiction, any user not checking each update it receives is vulnerable to the NSL + gag order famous combo in case of a targeted attack. I recognize that this is probably something that most people do not include in their threat model but I'm still a bit dubious about the fact that convenience related to release management and not having to worry about interoperability is worth accepting the risks linked to a unique delivery channel, especially for what could (and is widely thought to) be a completely secure IM solution. "Almost secure" is frighteningly the worse obstacle to "secure"...
I'm admittedly biased since I'm convinced that federation, multiple client/server implementations and multiple distribution channels are a requirement for a secure IM infrastructure (which is why my heart goes to Matrix nowadays).