Did raising downvote limit to 500 made any difference in unfair downvoting? If so, giving upvoting to more qualified people will also solve this for some time, means we can focus on measuring the qualification.
Maybe we should be able to mark individual comments as unfairly upvoted. Higher unfairly upvoted scores might decrease the value of future upvotes of voters on that comment.