I tend to down mod rude and aggressive people on here and I also up mod people who I think have been down modded unfairly.
I am not sure if I am alone with my way of thinking.
It only becomes abuse when people resort to karma bombing: downvoting a lot of comments by one user without reading them in order to subtract maximum karma. Fortunately we now have several levels of software to protect against that.
I have seen quite a few comments that were extremely insightful, and/or interesting that got downmodded due to an unpopular opinion. The reason this is unfortunate is not only that you tend to miss these (assuming that there is a higher probability that you read or think about comments that are rated higher, which I am surely not the only one that is guilty of) but also that it tends to promote groupthink . This is especially important on a forum like this where we are here to learn and share our thoughts, ideas and experiences for a very particular niche: Starting startups.
I have noted that comments that don't promote the "build it and they will come" view tend to be voted down. Since this site is primarily populated with hackers this is entirely understandable - it is human nature to think that your part of the project is the most important. But the reason we all come here is (I presume) to learn. And the things about which we know the least are the things where we have most to learn.
It is not only a question of abuse, but also a question of opening peoples eyes to issues, problems and points of views that lie outside their expertise, but which they will probably encounter in a startup. And this includes such diverse fields as marketing, financing and sales.
I am here to learn about stuff I didn't know already, and that is often outside my field. In return for this I will offfer my opinions in the fields where I may have something to contribute.
At the end of the day this makes us all better entrepreneurs. Because as anyone who has ever done a startup will tell you - you have to get everything right. Hacking, finance, sales, PR, marketing, hiring, etc.
So I think that the up and down arrows should not express agreement, but insightfullness or truth. Not opinion. That way I will be able to judge the validity of a comment in a field that I do not know well by its points. And hopefully learn something.
Once there, I'll stick a post on the front page and get downmodded into oblivion.