Yes, and I also do it, but very rarely and as an exception (and I am strongly in the position of "sniping weakens the system").
The issue is that it is possible that the censor may have misunderstood and missed something that was not clearly expressed.
I'd argue that in this case it's also in the interest of the community to downvote, since the comment adds more noise than it contributes to the discussion. In the end it's a ranking; if other comments are better worded and clearer, everything else being equal, they deserve to be higher up.