Users rarely deviate from the established upvote/downvote patterns. In fact, I'd go as far as saying many users don't even read the comments before voting.
When two users are having a heated argument, it's common for a third person to respond to the 'right' person with an innocuous comment and be heavily downvoted for it.