I will never down vote anything on HN, even if it's a crap comment. Let the good comments rise to the top. However, I noticed I've angered some kind of stalker who will down-vote anything I post, no matter how uncontroversial. What a way to spend your life...
I also upvote comments that are wrong or that I disagree with, if there are interesting replies to that comment. People seem to forget that those will get swept away as well.