zlacker

>>uyzstv+(OP)
Yes, sorry! We're investigating, but my current theory is we got overloaded because I relaxed some of our anti-crawler protections a few days ago.

(The reason I did that is that the anti-crawler protections also unfortunately hit some legit users, and we don't want to block legit users. However, it seems that I turned the knobs down too far.)

In this case, though, we had a secondary failure: PagerDuty woke me up at 5:24am, I checked HN and it seemed fine, so I told PagerDuty the problem was resolved. But the problem wasn't resolved - at that point I was just sleeping through it.

I'll add more as we find out more, but it probably won't be till later this afternoon PST.

Edit: later than I expected, but for those still following, the main things I've learned are (1) pkill wasn't able to kill SBCL this time - we have a script that does that when HN stops responding, but it didn't work, so we'll revise the script; and (2) how to get PagerDuty not to let you go back to sleep if your site is actually still down.

>>dang+zk
Crazy that Dang literally manages HN in his sleep!

We all knew that but I haven't seen any confirmation before this.

>>shlomo+Rq
failing to manage HN in my sleep is more like it

>>dang+vu
We all have our moments, and I personally consider HN to be “best effort”, almost like a volunteer project. I’m not certain I’m correct: but thats the optics I have so my expectations are adjusted to that.

So don’t beat yourself up please.

When I worked for “SaaS unicorn” we typically had multiple levels of escalation, and acknowledging would have done nothing because the alarm would continue firing until fixed. Not sure what’s changed in 15 years of ops, I had assumed it would be better now- I can’t imagine silencing an alert totally by acknowledging it- if its still occurring.

I’m totally fine with how you handled it, if anything I am thankful. But that seems to be a system I would improve if I had the time.

“mute” is different than “resolve” to me, and both should exist. (Where mute is an acknowledgement of an issue as ongoing.)

>>dijit+Ev
Yeah we don't exactly pay to be on HN, not much to complain about. I appreciate everyone who works on HN.

>>gianca+pI
We pay with content and with the fact that we attract the talent that eventually ends up powering ycombinator investment rounds.

>>jacque+Vw1
It’s ad-supported. Any post with comments disabled is definitely an ad. Probably a lot of the others are, too.

>>phanta+0B1
Your comment makes me realize that I consume HN differently than many others, because I've never seen a post with comments disabled and I've been around here for at least ten years. It's not that I don't think they don't exist — they obviously do because you're mentioning them. I've just never encountered one, primarily because I don't casually browse HN, ever. I subscribe to a pushbullet channel that notifies me when a post hits 500 up votes. That's it. The list of submissions on the home page (even on reddit) is just overwhelming to me so I use the pushbullet channel as a sort of community curated "best of" or "trending" signal.

Not to say that I don't procrastinate or waste time doing other nonsense. I can definitely spend a lot of time reading HN comments, as I'm doing right now.

Anyway,anyone who finds themselves with a problem with HN should try that out :)

>>abusta+yU1
> Anyway,anyone who finds themselves with a problem with HN should try that out :)

To be clear, I wasn’t complaining. Just pointing it out. Aside from any more speculative benefit to YC for running the site, the site does run outright ads.

>>phanta+2C2
Sorry, I didn't mean to imply you had a problem with ads. By "problem" I meant "if you find yourself procrastinating a lot" (not you specifically, but the reader in general)

Apologies for the misunderstanding