(I'm not saying HN should do exactly the same thing, but one example is Slashdot's system where a comment can get downvoted in a way that tags it specifically as trolling/offtopic/whatever - https://slashdot.org/faq/mod-metamod.shtml seems to describe it alright)
Downvotes are for material that does not support productive discussion, if it is productive to discuss a comment (even on a meta level like “why it is a bad comment”?), then downvoting is inappropriate.