(I'm not saying HN should do exactly the same thing, but one example is Slashdot's system where a comment can get downvoted in a way that tags it specifically as trolling/offtopic/whatever - https://slashdot.org/faq/mod-metamod.shtml seems to describe it alright)
I downvote for off topic, or for comments which detract from the conversation. I believe most others with the ability to downvote do the same.
Replying to off topic posts would be counter to the purpose of downvoting.