Up-vote score = sum(karma of up-voter)
Down-vote score = sum(karma of down-voter)
Score is displayed in both absolute and relative terms. Absolute score would be the same method as we're currently using. The relative score is presented as a part of the whole.
Something like [+++++++|--] could represent the ratio of the positive score to the negative score (which are the weighted scores based upon karma).
And, as a possibly added benefit, taking this approach enables the ability to reduce the karma level before allowing of down-voting, making people feel like they're able to participate more-fully earlier.