I am vain about my own karma though. Maybe a UI toggle to hide my karma so I would care less about it?
Or just make the user karma a simple average over all their posts and comments. Anything but a straight number that goes up each time!
I use my own karma display as a way to check if its likely i have new replies.
If the purpose of this site is exchange of ideas (rather than personal interaction), who wrote something should never matter. Only what was written matters. Discussion threads become collectively sourced arguments rather than ego battles. Take the identity of posters out of the equation.
In the rare cases where it matters (e.g. "Show HN" threads where the author offers to answer questions), it should be no problem for people to explicitly identify themselves ("author here"; this is mostly already happening anyway).
I think that would underscore that what appears to be a conversation between two people is not. That's important and I think often lost in online discussion. If UserX doesn't reply they might have been persuaded to UserY's position and one way of signaling that is to not reply. Meanwhile there's very likely someone who will read a well reasoned and backed up comment and nonetheless disagree, and they're most likely to reply given their disagreement. More so in the age of bots. The aggregate effect is for discussions to be weighted harshly negatively to the point of destructively.
Edit-I'm not very aware of the "famous user" effect. Sure there are usernames I recognize, but it's few, and the UI doesn't seem to give them much weight. I 'worry' more about the criticality of the site.
I'm tracking in my head a few hundred of users. I don't know the exact number because I never made an written list. Some users make consistently good comments in some topics, and it's an important signal for a discussion.
For example. ColinWright is a mathematician like me. I usually skim the math posts but he reads the whole post. So when he make a comment in that post it's usually accurate. If he says that in page 3, second paragraph there is a huge error, I just go to page 3, second paragraph and there is surely an error.
Nobody is perfect, but some users have earned a good reputations in some topics. I classified others as clueless enthusiastic, others as troll/morons/crackpots. Other are just unclassified. It's topic specific, so I may think a user makes good comments in one topic and regular comments in other topics. (I don't remember any case of good comments in one topic and really bad comments in other topics, but I have no formal list to check, it's just a fuzzy memory list in my head.)
We do need a way to distinguish participants so we can correlate replies to earlier comments of the same participant; otherwise it's way too disorganized as you don't know which voice is which.
Revealing usernames eventually is right, I think, because clicking through to see how that person describes themself (job, hobbies, etc) is an interesting dimension to their comment.
Also, for better or worse, I think people put more effort into making things (including written comments) that are attached to their identity in some way, so usernames increase quality.