(I'm not saying HN should do exactly the same thing, but one example is Slashdot's system where a comment can get downvoted in a way that tags it specifically as trolling/offtopic/whatever - https://slashdot.org/faq/mod-metamod.shtml seems to describe it alright)
I downvote for off topic, or for comments which detract from the conversation. I believe most others with the ability to downvote do the same.
Replying to off topic posts would be counter to the purpose of downvoting.
So yes downvotes for mere disagreements are okay here.