By using phone numbers as IDs signal can rely on your phone's local contacts (meaning they don't have to send your social graph to their servers). This way they can keep very little metadata on you.
There's pretty much nothing for them to turn over except the fact that your phone number has the signal app.
Most of the other secure apps could turn over your entire contact list (which could be damaging for people in a protest that are being targeted).
Confirming a single phone number has the app is not nearly as big of a deal (I'd argue it doesn't matter at all).
There's nothing inherent in phone numbers here. Both iOS and Android also allows you to add e-mail addresses (and other identifiers) to your local contacts. I'm yet to hear an argument as to why e-mail addresses or other identifiers can't be used in addition to phone numbers, or why it would be a complicating factor.
I think they are working on non-phone number IDs though (Moxie was in an earlier signal thread on HN recently and mentioned it).
It will take the authorities a bit more time (i.e. someone throws away their burner phone and authorities hack it)(with the assumption that phone numbers/SIM activations are provided using valid ID as it happens in many countries).
https://en.wikipedia.org/wiki/Global_surveillance#Infiltrati...
Edit: And that from netzpolitik (highly trusted german source) under 'A global passive adversary' that's the interesting part: https://netzpolitik.org/2017/secret-documents-reveal-german-...
There is also I2P network, which is even harder to break (unless someone owns practically all nodes there).
And no you can trace it thru the ISP's, the problem is the latency, Connection from here to there in that millisecond trace one...and so on.