(I'm not saying HN should do exactly the same thing, but one example is Slashdot's system where a comment can get downvoted in a way that tags it specifically as trolling/offtopic/whatever - https://slashdot.org/faq/mod-metamod.shtml seems to describe it alright)
I'm reminded of the xdcd "someone is wrong on the internet!" comment. I'm starting to get to the point where I don't feel like arguing or even explaining things; I just want to downvote, move on, and hope my moderation contribution has made the shape of the thread more useful to people.
If I had to explain my downvotes every time, I'd hardly ever bother, and I suspect that's true for most people. A forum where no one wants to take the time to moderate is not going to be a healthy place.