Definitely the members IMO; a moderator can snipe out the occasional troll, but if all members are being pricks there's nothing that can be done about that. What you need is to maintain an atmosphere, a culture, etc. You need the community to call one another out and keep one another accountable. And you need to nip any broken windows in the bud - Reddit's comment threads often spiral out of control and into a spammy mess of memes and references for example, simply because that's part of their culture. It's harmless enough on Reddit, but if that happened on HN the comments section would diminish greatly in value.
One of the other forums I frequent would be chock full of pricks by HN standards. They are known for being jerks on other forums that cover the same interest. It's still a very good community as long as you don't take everything personally.