I downvote egregious misinformation, but not honest mistakes. A correction in a comment is more appropriate in those cases. Disagreement and differing opinions do not get downvotes from me - they are part of what makes the discussions interesting. But comments that are actively rude or try to take people down a peg do get my downvote. So do people who clearly are just pushing a personal agenda unrelated to the current discussion. I sometimes try to picture the comments as a real conversation... if you would have said the same thing in a group sitting together real life, would that have been a comment that fit into the conversation? Or would it make the group give you a 'WTF?' look and slowly back away?
At the same time, when I receive downvotes, most of the time they are correct - if I post a comment when I'm in a bad mood, my tone gets snarkier and I get downvoted more often. And that is fair. So I tend to just accept them, move on, and try to be more aware of my communication habits in the future.