Unless you do it to the outgroup. Then it's fine! Laudable even!
Same as shutting down journalists and other accounts. It was nothing to fret about when the opposite side used to do it, "they were misinforming or borderline bad anyway, and they could always start their own blog or something, so it wasn't censorship" and so on.
It's also comic: pundits pissing on free speech (tons of cheering when people were cancelled before, and lots of articles on how it's justified and free speech is not the be all end-all) making a u-turn to call for free speech and condemn Musk's account shutdowns now, while Musk and co that was defending free-speech before is now censoring accounts, while the "free speech" proponents in the previous round are now cheering him for it...