The idea that large public internet forums inevitably degrade has been the default understanding of internet forums since before PG started HN—in fact he started HN as an experiment in escaping that fate (https://news.ycombinator.com/newswelcome.html). So another way of putting this is that you think HN's experiment has failed. That's fine, but I cling to a different view for the time being; out of cognitive dissonance if nothing else, since I spend days and nights working on it.
The guideline about flagging egregious comments is there to prevent obviously awful comments from generating off-topic, repetitive flamewars. If a comment like that is flagged and downvoted, that is challenging it. It means the community has rejected it. Better still, it minimizes its influence by stopping it at the root. Responding by pouring fuel on the flames is what causes it to expand and grow. Since you refer to not feeding trolls, you obviously know this. Beyond that, I'd have to see specific examples.
Since I don't know what semi-private controlled forums you're referring to and can't look at the criticisms of HN people are making there, it's impossible for me to evaluate them. That's a pity, because we might be missing opportunities for improvement. But the fact that they're starkly negative doesn't say much by itself. Smaller communities always have a negative view of larger communities—that's how community identity gets created. And cohesive communities always have a negative view of non-cohesive communities, because divisive topics inevitably produce responses that fall outside their acceptable spectrum. Sharing an acceptable spectrum is part of what makes a community cohesive. We don't have that on HN, certainly as a function of size, and probably also for other reasons.
And to call back to my original examples: The Katie Bouman thread(s) [1] [2] and the Women: Learn To Program [3] threads should give you quite a lot of pause. The fact that such benign incidents resulted in large flamewars is specifically an issue because it indicates a greater rot growing inside HN's culture.
There's obviously more examples (such as anything politically related almost immediately devolving into flamewars or whataboutism) which indicates that the mission statement simply isn't working. And the greater problem of flagging only works if the community as a whole agrees in a positive direction; if suddenly tomorrow HN was filled with people who held highly negative beliefs, then the flagging system fails.
When I look at other forums such as SomethingAwful, Penny Arcade etc I see them as surviving because they have much stronger moderation while maintaining a sustainable community size. And right now I don't see HN outlasting either of those communities. Without some sort of cohesion guiding the community, the end result is that the site will eventually be pulled away from its original purpose.
To sum up what I think would be necessary:
1. Long time contributors would need to be emphasized more. Especially the high quality contributors, because they serve as a way of keeping a community united.
2. The mission statement of HN needs to be less vague and more to the point. Keep a focus solely on things that happen with the tech community and issuing harsher but smaller punishments to people that cause issues. You issued a warning to me a while ago because I was being an ass, and on other sites a warning like that would've resulted in a harsher punishment like a temporary probation.
3. Politics is inescapable as was found out during the 'political detox' week. But other forums can help moderate and control political debates and inflammation by keeping them solely inline with the site's mission statement (ie: a gaming site focuses on politics as it relates to games).
That said, implementing a lot of this might be almost impossible at this point because people would decry censorship almost immediately, resulting in a large reactionary wave. Which unfortunately I think also says a lot about the overall lassiez-faire moderation style HN employs for everything but the most egregious and repeat of offenses.
[1] https://news.ycombinator.com/item?id=19632086
> The fact that such benign incidents resulted in large flamewars [...] indicates a greater rot growing inside HN's culture.
Such incidents result in flamewars because society is polarized on these topics and getting more so. Is there a single place on the open internet at HN's scale or greater that is any different, or indeed isn't worse? HN can't be immune from macro trends. (For example, there have lately been more nationalistic flamewars, especially about China. That's plainly related to shifts in geopolitics.) If HN is a ship, the sea is stormy. We can't control the waves, or how much vomiting the passengers do. If we focus on what we're actually able to affect, maybe we can prevent the ship from sinking.
I took another look at the threads you linked to and don't see what you see. The balance of the community there is clearly supportive. Most of the indignant comments are from people protesting against the negative reactions, which were clearly in the minority. Those don't represent the community, although (as always) the community is divided. So I come back to what I said in my first reply to you: if you're judging the community by the worst things that appear here, that's a fallacy. (Actually, I'm talking about your links #1 and #3. #2 was worse.)
Perhaps my standards are lower than yours? That's possible. On the other hand, sometimes when people post complaints like yours I have the impression that what they really want is for us to take their side on every issue and ban everyone on the opposite side. We can't do that. The community would not allow it, and trying to force it would destroy it—what good would that do? There's a deeper reason too: enforcing homogeneity would be incompatible with intellectual curiosity, and we're optimizing for the latter. The price of that is a certain turmoil on divisive topics—enough to convince ideologically committed users that the site is dominated by the other side (see "hostile media effect" above). If you don't think people on the opposite side of the issues have just as "starkly negative" a view of HN as people in your circles do, I have a long list of links I can share. In fact I almost unearthed them to post here, but decided to spare you.