I'd be willing to bet that if you could somehow run an experiment in parallel where you had one Reddit with real bans, and one with soft bans, the quality and nature of interactions on the soft ban one would be much, much worse even outside of banned communities.
This is sort of talking around an argument. You could say the same thing about a subreddit dedicated to re-electing a local alderman because of his policy on the maintenance of public parks. Speech is meant to inform, or to affect change.
The question is whether you're going to use an online annoyance argument to moderate controversy on a platform. If the justification for why you're going to moderate speech is that people who are not annoyed by that speech might react to it, you've moved squarely into making "genuine arguments for true censorship: that is, for blocking speech that both sides want to hear."