I've thought a lot about why, since I used to really enjoy HN - now it's just one of a few newssites I visit every day. It's hard to quantify but here are my reasons and my take at the decline:
1) The obvious one: Signal to noise ratio in the comments is way down. The problem is twofold - there are both more bad comments, and the ones that are good aren't necessarily voted to the top. This makes it harder for me to find the nuggets that would be shown at the top of every comments page a year or two ago. As others have pointed out it sound easy but is in fact a very hard problem to solve.
2) The interaction in the comments is less interesting. I used to have great arguments in the comments. Sometimes I would convince someone of my point of view sometimes it was the other way around, sometimes there just wasn't agreement to be found. But it was always interesting and civil, and I very often learned something new. Engaging in, and watching others have interesting discussions was for me one of the main things I loved about HN. It's like when you go to a dinner party and get to sit next to this incredily interesting guy that is exceptionally insightful and has some really interesting things to say. The conversation leaves a mark on you.
3) I often find that the comments I make that I personally find insightful or interesting don't get a lot of upvotes, while the ones that state something obvious or funny get more upvotes. This isn't encouraging me to interact with people here on an intellectually interesting level. If others do this as well, which I suspect they will, then it's extremely degrading to the discourse in the comments. I often find that I don't bother to write up a response to something because I know won't get a lot of attention. Sometimes my points are totally missed.
4) Maybe I've outgrown the site. Many concepts that were new to me when I joined HN are now familiar, and many discussions have already been had. RiderofGiraffes describes it well in the linked comment.
I owe a lot to HN, and I really want it to succeed, so I stick around and hope that things will change. But for now it's from a less engaged position.
Now, maybe it's just me, but I used to like the science-type stories, or other stories that taught me something interesting and novel from some branch of human knowledge. But I just checked the first 90 stories and there's nothing matching that description. Instead there seem to be an awful lot of "gossip" and "personality" type stories. Tesla vs Top Gear! Tech CEO shoots elephant! Trollish "What I hate about facebook" stories! The interminable "Is it a bubble?" discussion!
On the other hand, it might just be my opinion... obviously somebody is interested in the current front-page stories or else they wouldn't have been voted up. Do other folks think that the interestingness of the stories has declined?
One possible solution: depend more on credentials, and give people who have useful things to say / special background knowledge priority. Weigh their votes more and give them karma bonuses. Someone from a YC startup might be given more weight than someone who opines on large-scale, amorphous social problems from a generic position ("outsourcing: economic doom or natural phenomenon? Let's flame!").
I think the big thing, more than anything else, is the learning aspect. We tend to learn more from people who know a lot of stuff and have unusual experiences or abilities. Those people tend to cluster, then less smart people cluster around them, eventually leading toward decline. One way to counter that: identify those people and give them a louder voice. I don't think there's a technical way to do this that won't be gamed, unfortunately, which leads to a major scaling problem.
I mostly try to follow this rule: I mostly comment on areas related to schools / teaching / universities (I'm a grad student in English lit at the University of Arizona), books, or literature. I try to make comments that are as rooted as fact in possible; for example, when someone a while ago said there were no laptops with IPS screens anymore, I left a link to an Anandtech review of one and didn't say much else. Problem is, I don't have any good way of systematizing that or rewarding it except on an individual level.
I always found the boards I enjoyed best back when I ran a C-64 BBS (yes, that was a BBS running ON a C-64 with a 1200 baud modem and two floppy drives) were the ones with appropriately benevolent "dictators" who used a light touch to keep things real and on track. I can't say I've seen anything on the internet to convince me that there's been a notable improvement on such.
For me the only solution is to allow down-voting submissions by top contributors - because the lack of good comments is a byproduct of the lack of good submissions..
Isn't this the role of Slashdot's editors? Or do they choose the stories as well? Either way, Slashdot has almost no interesting stories these days.
This means that the community-at-large is effectively given the opportunity to vote on a sub-optimal selection of the submitted stories.
This can perhaps be fixed by briefly flashing all new stories towards the bottom of the top news page, to give them a brief chance at wider exposure. One way to do this would be to make a queue, and put every new story on the bottom of the front page for 5 minutes barring a certain number of down-votes for moderation.
This might really effect which stories actually end up on the top news page (though perhaps only for stories submitted during peak times).
A better solution than "giving the smarter people a louder voice" would be to "give the louder people a smarter voice."
My suggestion below is to add a good way to discreetly provide feedback. Encourage people to send messages to you about what they think of your post, with the goal of encouraging everyone to improve the quality of their posting, and also to be more thoughtful about up/down voting.
My main thought every time this topic comes up is "Community quality don't maintain itself."
So vote on comments you like or don't like, every time.
Check the "new page" and vote up stories you like. I know very few people do this, because I find good stuff overlooked there all the time.
And submit stories you want to see here. Someone has to do it or they'll never show up here.
One specific instance was when I submitted an XKCD shortly after joining. I eventually realized it wasn't really what people here were looking for. When elections rolled around, I submitted a Wapo editorial that was really off-base, and the reaction I got in response was deservedly negative.
However, in this process of learning to be a more civil and well-reasoned person (which I do credit a lot of to the good folks here on HN), I've always found a measure of grace. I don't think the reactions I've gotten for anything I've submitted or commented with have ever been really mean. Criticism has always been positive. This is the main aspect of HN, besides being intellectually stimulating, that I've come to appreciate most. It has really helped me to grow (I'm a college sophomore now and I really am grateful to the kind people here who've taught me so much since high school).
I agree with pg's general observation that things seem to have become a bit meaner and less constructive. I'm not sure how to fix the front page, but the type of comments I find representative of the old HN are informative, well-reasoned, and forgiving comments. I entirely agree with your bit about "I disagree" comments, though I don't think it would be as much of a problem if it were done a little more nicely. It's okay to disagree with an article, but the tone that some people take, mehh. That said, I almost always read the comments prior to reading the article, because I count on the contrarians to shed some light that I might otherwise be ignorant to.
I think everything would be okay if everyone just reminded themselves to be civil and not to use downvotes to disagree.
I'd actually been considering hiring someone to run HN, though not to moderate it in quite as hands-on a way as you're suggesting. Interesting idea though.
Emotionally charged rants get attention. Careful reasoning gets eviscerated by pedants. So people are learning to write for the audience.
I find it really odd that you cant downvote until 500 karma, and it appears you can never? downvote stories?
And then there are comments in this thread that say "its a really hard problem to figure out" -- no it isn't, you just need to have faith that moderation power will be used appropriately by the site's audience and give them the appropriate ability to do so.
Just like on quora, they seem to have blinders on to systems that work because they want to believe that only their design team could possibly come up with something novel and in that novelty find the best solution.
The later parts of the book (chapter 8 and beyond) seem applicable to the problems HN is facing.
PM me if you want, I'll send you my copy of the book.
On the topic of moderation I would suggest the ability to somewhat "follow" good commenters, and even perhaps block bad submitters. This way, reputation is not only karma, but quality of followers.
This is the biggest one IMO.
Be warned, he's going to strongly advise you to set up a backchannel for public metadiscussion. Jeff Atwood resisted it vehemently at first when another metafilter moderator was a guest on his podcast, but they did eventually set up their meta subsite in the same vein.
Without trying to insult them, I know that it is a very close-knit and strong community, but the intellectual barrier is very low. The comment quality is very low. The notion that there is anything there that can help us is likely being spurned because, right or wrong, that's not what we're striving for.
While I was a little shocked to hear PG clarify that HN is a place for hackers and not startup founders (I kind of expected the opposite,) I think we can all agree that this isn't meant to be a site for the lowest common denominator.
I personally believe that quality control starts with the submissions. The higher quality submissions, the higher quality comments they'll attract. But that's just me, apparently.
It is a good filter for low quality content, but it also filters out things I find interesting.
I dont follow.
I am saying that having a better system for your community to converse allows the community to thrive.
You may think the intellectual barrier to entry is low, or the intellectual quality of what you read on reddit is low - and I would counter that maybe your looking at the wrong subreddits for you.
There are a ton of subs that I cant stand, dont read, dont follow etc...
I simply ignore them - but I am talking about the mechanics of the forum. The ability to have ongoing conversations, up/down vote comments and submissions and the ability to categorize content via the submission process.
You're talking about the community being beneath some HN standard, where I am talking about the characteristics of reddit's infrastructure that allow the community to interact.
For some reason, people think that we need automated tools to weed out those that would be beneath their interests, but I don't think that is possible.
Also, how would you accommodate people who are just learning, trying to learn or are experts in other subjects yet are trying to expand their base? You cant say that the intellectual barrier is too low such that people who don't know what we are talking about cant participate...
Actually, yes, I am. I firmly believe, for better or worse, that the more popular a site becomes, the more interests it has to cater to. Somebody else posted in the thread that the three options for a community to stay true to its roots is to either exclude newcomers, aggressively moderate newcomers, or succumb to newcomers.
Allowing everybody an equal share participation on the moderation capabilities panders to the latter. If everybody can moderate out, and the site continues to grow, then I think it's only a matter of time before the harder-to-understand articles are filtered out in favor of more easily digestible pieces that appeal to a broader majority.
I think the main difference between here and the notion of subredditing is that effectively, HN IS a subreddit, targeted at Hackers. I'm obviously not the say-all authority, but I suspect that splintering it further is not only unnecessary, but runs contrary to the concept of a focused community.
In fact, that there are subreddits that work so well (again in my opinion) speaks primarily to the fact that subreddits aren't very easy to find or discover.
As for the people who are just learning? I honestly don't know how to handle that. Though HN has worked really well with a diverse group of experts. There are lawyers, doctors and professionals of all sorts that are able to contribute to the areas they are knowledgeable in -- the trick is in somehow enforcing the restraint for them to not comment on things they know little about, or at least to not comment unintelligibly. The more aware of the community ideal everybody is, the more easy that becomes; but the more diluted the population grows to be, the harder that becomes to enforce.
Like I said, I don't know all the answers, and I certainly don't mean to impugn anywhere else in favor of HN - Reddit has plenty of perks, I'm sure, but it is the rare community that is able to subsist through popularity. The closest that I can think of to have lasted is kuro5hin, and they certainly had their own communal warts as well.
However, we are talking about how the scale of HN is getting to the point where people are complaining about S/N ratios and as you say:
>exclude newcomers, aggressively moderate newcomers, or succumb to newcomers.
I think this is shortsighted, elitist or both. First, you/whomever posted this has clearly left out a better option; ADAPT to newcomers/scaling.
While I agree that HN may have been like a subreddit, with a target audience, HN is growing, entrpreneurship is growing, the startup ecosystem in the valley is growing, our knowledge base is growing.
Eschewing newcomers and growth is to operate in fear of progress.
Taking the spill-over and iterating what HN is to accommodate is not to "lose its cred" so to speak...
Further, I am not advocating a straight adoption of reddit, but I do feel that HN can learn a lot from how they enable the community.
I had written a bunch of suggestions in my first reply, but deleted them, but I was suggesting some options along the lines of:
Validate SMEs in given areas and give them high-level moderation/influence.
Post a clear HN-etiquette that delineates posting format/commenting format that the stie wants.
Earn further features/user abilities through karma - such as allow SMEs/high karma users to only create HN-subs.
There are a lot of things we can do, but to cry in the corner over newcomers is never a winning solution.
I'm not suggesting that we do, obviously. However, newcomers have the disadvantage of not knowing the history, the ethics, the goals/ideals. They only know what it was like 'when they joined'. When I joined, I was corrected a few times for missteps I made. The growth rate makes that less and less feasible. It isn't that I hold a grudge to new people, they are just disadvantaged when it comes to the communal etiquette standards.
So I generally think that good stories shouldn't decay as fast as they do. And maybe also when there is some activity on them.
Maybe also we should have some category like "best stories of yesterday", putting some fresh attention to the good ones.
First, you say that you don't recognize that behavior is influenced by things like up-votes, down-votes, and deep threading. In the next breath you argue that a better system has a positive impact on the community. I don't see how you can have one without having the other. Making a community “thrive” is changing it. If you disagree with this please explain to me why communities like thriving, but don't like not thriving. If they are the same than they shouldn't care either way. If things have changed than you agree that the changes had an affect. Also a better system being better is something that I find to be true by definition.
Next you argue for downvotes, by saying that reddit has proved itself in virtue of subreddits. I don't think this is the case. I think that subreddits are harder to get to, creating a selection-bias in which only the people who are interested in that subreddit end up contributing to it. In other words I think you can explain the success of reddit's subreddits with subtle elitist bias. This creates a huge amount of irony, since your post is actually arguing against elitism.
You also claim that some people think we need automated tools to weed out things that our beneath are interests. This is another point that is sort of funny. We not only need those tools, we have them [1]. They are built into our minds.
In the end you ask rhetorically: can you ask someone who doesn't know what your talking about to not contribute to your discussion? This backfires too. Plato said: “Wise men speak because they have something to say; Fools because they have to say something.” Proverbs continually tells the fool he should shut up (I'm almost done, really). The idea that you can ask people who don't know what they are talking about to be quiet isn't new or immoral.
I'm probably misinterpreting you, but that irony made my brain happy and I wanted to share it.
I wonder if it wouldn't make sense for the front page to have a few "new" articles show up as well. Like, sprinkled throughout the list, or maybe just show three of them at the top. Get them in front of the whole audience, rather than just the small subset of users that visit the "new" page regularly.