I tend to down mod rude and aggressive people on here and I also up mod people who I think have been down modded unfairly.
I am not sure if I am alone with my way of thinking.
It only becomes abuse when people resort to karma bombing: downvoting a lot of comments by one user without reading them in order to subtract maximum karma. Fortunately we now have several levels of software to protect against that.
The karma system should predict how good I am going to feel after consuming the media -- nothing more. The problem is that for a simple question, the solution ain't so simple. I believe there is one, and I believe we've danced around this subject long enough for those paying attention to have figured it out.
Also -- thanks for the term "karma bombing". I've been a victim in the past, now I have a cool geeky name to describe it.