I tend to down mod rude and aggressive people on here and I also up mod people who I think have been down modded unfairly.
I am not sure if I am alone with my way of thinking.
It only becomes abuse when people resort to karma bombing: downvoting a lot of comments by one user without reading them in order to subtract maximum karma. Fortunately we now have several levels of software to protect against that.
I guess we don't always behave in a grown up way online. I am as guilty of that as the next guy.