The time sink factor is also obvious. However, I tend to swing back and forth between very productive and not very productive. When I am in a coding mood I rarely browse HN or comment and instead I hack like crazy for hours every day. When I am burned out or feeling like I need a break from chasing some bug I'll stop by here and read.
So the time sink aspect is doable for me at least. No one can work constantly, without any breaks, and maintain their sanity.
I disagree with HNers(mostly because I offered radical opinions) all the time. Sometime I get downvoted, sometime I got lot of points. There will alway be assholes who downvote people for simply disagreeing but they're outweighed by people who vote me up for adding to the conversation.
That's too bad, because you can.
Stop worrying about your karma.
Naturally the reverse is also true and you can't help but not want either number to fall. In my case at least these feelings are involuntary, despite the fact that I know these are pretty much worthless numbers. (Except for the fact that karma average does effect the weight of your votes.)