Why are mean comments upvoted? Answer: I don't know.
People learn how to act on HN by watching what gets upvoted, listening to the tone of discussions, and reading the submitted articles. Presumably, the unwanted comments are being made by new members of the community. Somehow, these new members were not properly taught by the community. In which step were they not properly taught?
I would like to submit three possible problems, along with three possible solutions.
1) The problem is in the voting system. Mean comments are being upvoted, and the senior members of the community are largely powerless to stop these comments. Sure, they can downvote, but they are just one vote, and there are still many more junior members who will upvote the mean comment. If you believe that: Senior members know what's best for the community, these members are senior because they have high karma, these members have high karma because the community has voted that these people know best. Weighing a vote by the karma of the user who made that vote would solve the problem of mean comments being upvoted.
2) The problem are the stories that make it to the front page. Mean comments and the votes they receive are a symptom. The users who upvote are getting their social cues from the stories they read on the front page. Broad requirements on stories that are HN worthy allow for a wide variety of stories to get posted to HN. This is good for somebody who sifts through the 'new' section, but it also means that the only stories that get massively upvoted are stories that have general intersections between all of our interests. Evidence seems to show that the most common shared interest is gossip, which is conveniently unwanted by the community. The solution in this case, is to make stricter requirements about what stories are allowed.
3) The problem is that bad apples will always exist no matter what you do. At the moment, the easiest place for bad apples to exist is on the front page of HN. Unfortunately, this is also the place a lot of normal users like to exist. Perhaps a sandbox could be made for the bad apples to hate each other, and allow the normal users to exist in separate but equal lives. Unfortunately, this seems to go against the HN spirit, and I can't think of any useful ideas on how to implement such a sandbox without it sounding like a subreddit.
Finally I would like to add: I like that HN takes the time for these self analysis every now and then. But, I think it's important to remember that we don't know what's best for us. The mere fact that we will upvote the type of content we don't want shows this.
This leads me to reiterate a comment best stated by idoh: "Let us not be too hasty in proposing solutions when the problem isn't really understood. At best they are shots in the dark. Even after you ship them you wouldn't be able to tell whether the fixes actually did anything or not."