I'm happy to (attempt to! re-)upvote comments that have been downvoted, as long as they're actually thoughtful and contribute to the argument/conversation, and obviously (and often!) I'll upvote a well thought through comment.
I seem to go through periods where I downvote more often, but almost without exception it's just downvoting pointless/sarcastic/unpleasant comments. Gotta say though - for a 10+ year old community, I remain amazed that the discourse is usually civil, largely intelligent and still(!) remains spam free.