I've thought a lot about why, since I used to really enjoy HN - now it's just one of a few newssites I visit every day. It's hard to quantify but here are my reasons and my take at the decline:
1) The obvious one: Signal to noise ratio in the comments is way down. The problem is twofold - there are both more bad comments, and the ones that are good aren't necessarily voted to the top. This makes it harder for me to find the nuggets that would be shown at the top of every comments page a year or two ago. As others have pointed out it sound easy but is in fact a very hard problem to solve.
2) The interaction in the comments is less interesting. I used to have great arguments in the comments. Sometimes I would convince someone of my point of view sometimes it was the other way around, sometimes there just wasn't agreement to be found. But it was always interesting and civil, and I very often learned something new. Engaging in, and watching others have interesting discussions was for me one of the main things I loved about HN. It's like when you go to a dinner party and get to sit next to this incredily interesting guy that is exceptionally insightful and has some really interesting things to say. The conversation leaves a mark on you.
3) I often find that the comments I make that I personally find insightful or interesting don't get a lot of upvotes, while the ones that state something obvious or funny get more upvotes. This isn't encouraging me to interact with people here on an intellectually interesting level. If others do this as well, which I suspect they will, then it's extremely degrading to the discourse in the comments. I often find that I don't bother to write up a response to something because I know won't get a lot of attention. Sometimes my points are totally missed.
4) Maybe I've outgrown the site. Many concepts that were new to me when I joined HN are now familiar, and many discussions have already been had. RiderofGiraffes describes it well in the linked comment.
I owe a lot to HN, and I really want it to succeed, so I stick around and hope that things will change. But for now it's from a less engaged position.
One specific instance was when I submitted an XKCD shortly after joining. I eventually realized it wasn't really what people here were looking for. When elections rolled around, I submitted a Wapo editorial that was really off-base, and the reaction I got in response was deservedly negative.
However, in this process of learning to be a more civil and well-reasoned person (which I do credit a lot of to the good folks here on HN), I've always found a measure of grace. I don't think the reactions I've gotten for anything I've submitted or commented with have ever been really mean. Criticism has always been positive. This is the main aspect of HN, besides being intellectually stimulating, that I've come to appreciate most. It has really helped me to grow (I'm a college sophomore now and I really am grateful to the kind people here who've taught me so much since high school).
I agree with pg's general observation that things seem to have become a bit meaner and less constructive. I'm not sure how to fix the front page, but the type of comments I find representative of the old HN are informative, well-reasoned, and forgiving comments. I entirely agree with your bit about "I disagree" comments, though I don't think it would be as much of a problem if it were done a little more nicely. It's okay to disagree with an article, but the tone that some people take, mehh. That said, I almost always read the comments prior to reading the article, because I count on the contrarians to shed some light that I might otherwise be ignorant to.
I think everything would be okay if everyone just reminded themselves to be civil and not to use downvotes to disagree.