I've thought a lot about why, since I used to really enjoy HN - now it's just one of a few newssites I visit every day. It's hard to quantify but here are my reasons and my take at the decline:
1) The obvious one: Signal to noise ratio in the comments is way down. The problem is twofold - there are both more bad comments, and the ones that are good aren't necessarily voted to the top. This makes it harder for me to find the nuggets that would be shown at the top of every comments page a year or two ago. As others have pointed out it sound easy but is in fact a very hard problem to solve.
2) The interaction in the comments is less interesting. I used to have great arguments in the comments. Sometimes I would convince someone of my point of view sometimes it was the other way around, sometimes there just wasn't agreement to be found. But it was always interesting and civil, and I very often learned something new. Engaging in, and watching others have interesting discussions was for me one of the main things I loved about HN. It's like when you go to a dinner party and get to sit next to this incredily interesting guy that is exceptionally insightful and has some really interesting things to say. The conversation leaves a mark on you.
3) I often find that the comments I make that I personally find insightful or interesting don't get a lot of upvotes, while the ones that state something obvious or funny get more upvotes. This isn't encouraging me to interact with people here on an intellectually interesting level. If others do this as well, which I suspect they will, then it's extremely degrading to the discourse in the comments. I often find that I don't bother to write up a response to something because I know won't get a lot of attention. Sometimes my points are totally missed.
4) Maybe I've outgrown the site. Many concepts that were new to me when I joined HN are now familiar, and many discussions have already been had. RiderofGiraffes describes it well in the linked comment.
I owe a lot to HN, and I really want it to succeed, so I stick around and hope that things will change. But for now it's from a less engaged position.
Now, maybe it's just me, but I used to like the science-type stories, or other stories that taught me something interesting and novel from some branch of human knowledge. But I just checked the first 90 stories and there's nothing matching that description. Instead there seem to be an awful lot of "gossip" and "personality" type stories. Tesla vs Top Gear! Tech CEO shoots elephant! Trollish "What I hate about facebook" stories! The interminable "Is it a bubble?" discussion!
On the other hand, it might just be my opinion... obviously somebody is interested in the current front-page stories or else they wouldn't have been voted up. Do other folks think that the interestingness of the stories has declined?
For me the only solution is to allow down-voting submissions by top contributors - because the lack of good comments is a byproduct of the lack of good submissions..
I find it really odd that you cant downvote until 500 karma, and it appears you can never? downvote stories?
And then there are comments in this thread that say "its a really hard problem to figure out" -- no it isn't, you just need to have faith that moderation power will be used appropriately by the site's audience and give them the appropriate ability to do so.
Just like on quora, they seem to have blinders on to systems that work because they want to believe that only their design team could possibly come up with something novel and in that novelty find the best solution.
Without trying to insult them, I know that it is a very close-knit and strong community, but the intellectual barrier is very low. The comment quality is very low. The notion that there is anything there that can help us is likely being spurned because, right or wrong, that's not what we're striving for.
While I was a little shocked to hear PG clarify that HN is a place for hackers and not startup founders (I kind of expected the opposite,) I think we can all agree that this isn't meant to be a site for the lowest common denominator.
I personally believe that quality control starts with the submissions. The higher quality submissions, the higher quality comments they'll attract. But that's just me, apparently.