This would make people think twice before down-voting. It would also reduce "knee-jerk" up-votes, as people would be more likely to reserve up-votes for things that truly deserve it.
It would also mean that long-time users will naturally do more of the moderation, as they have lots of points and may not be as afraid to use them.