There's an obvious conflict of interest in reducing another user's karma risks their ability to also downvote. (Of course, just because it's imperfect doesn't mean it's a bad system.) If this is the intended usage then perhaps it would be fair to expand downvoting rights to other users?