If you want to downmod, it's no big deal. But there seems to be some proportion of the audience that feels vindicated in their views just because the opposite view was downmodded. Downmodding does not make opposing positions less legitimate.
These people seem to be different than the ones who write replies. I've had multiple lengthy threads where I hardly get any mods either way while discussing, and then when I wake up the next day I have -20 karma, but no new replies.