I think vast majority of downvotes that I see on HN are more or less like that. Absolute trolls and disrespectful ones are on top of the list.
Usually the ones that people complain the most about are off-topic comments that get downvoted. I see them specially happening when the post itself is also kind of off-topic. Those posts tend to be open ended discussions. And if you are late to the discussion it's possible that the discussion is already headed to a more specific sub-topic. Some new comments are then considered off-topic by some users.
Upvote is more powerful. I personally try to use upvote to keep what I believe is relevant on the top. And that's what I see most of the users do as well. But, I also don't mind if my comment get downvoted. There is nothing negative about downvotes. It's just other users telling me that this is off-topic right now.