The rationale is that to me, there's quite a difference between a comment that has 1 karma because there was no upvotes/downvotes and one that has 1 karma because there was 20 upvotes and 20 downvotes.
So, we could have something like:
5-3=2 points by username