When Signal finally drops SMS support, I will also drop Signal. It always was SMS foremost, secure communications second when I initially started using it. Having people on Signal was an added benefit, not the main drive for using it.
For me personally, Signal will turn into a messaging app among dozens of similar apps, of which I have enough already. SMS was why I stuck with Signal over the years I used it and now I have no real reason to stay on it.