(I'm not saying HN should do exactly the same thing, but one example is Slashdot's system where a comment can get downvoted in a way that tags it specifically as trolling/offtopic/whatever - https://slashdot.org/faq/mod-metamod.shtml seems to describe it alright)
I don't think we should make it any more complicated. You might benefit (on the margin) from some feedback, but that makes the conversation more noisy for everyone.
I can say that personally, if I downvote something without also replying to it, it's either because I don't have time to write a helpful response, or because I know my own contribution would make things worse.