I downvote for off topic, or for comments which detract from the conversation. I believe most others with the ability to downvote do the same.
Replying to off topic posts would be counter to the purpose of downvoting.
Some comments can go yo-yo, but I don't think it is possible to see the two counts separately.
So yes downvotes for mere disagreements are okay here.