I tend to down mod rude and aggressive people on here and I also up mod people who I think have been down modded unfairly.
I am not sure if I am alone with my way of thinking.
It only becomes abuse when people resort to karma bombing: downvoting a lot of comments by one user without reading them in order to subtract maximum karma. Fortunately we now have several levels of software to protect against that.
Whether it's worth the added complexity, I'm not sure, but it would resolve an ambiguity in the signals sent to commenters.
(To squeeze into the existing UI and mental model of this sort of site, I might make the current up/down mean quality -- should more people see this or should it recede from view? -- and add a right/left for agreement, giving posts tiny little inline spot polls, of a sort. Maybe it could even be a sparkline or two-tone percentage-bar.)