zlacker

>>uyzstv+(OP)
Yes, sorry! We're investigating, but my current theory is we got overloaded because I relaxed some of our anti-crawler protections a few days ago.

(The reason I did that is that the anti-crawler protections also unfortunately hit some legit users, and we don't want to block legit users. However, it seems that I turned the knobs down too far.)

In this case, though, we had a secondary failure: PagerDuty woke me up at 5:24am, I checked HN and it seemed fine, so I told PagerDuty the problem was resolved. But the problem wasn't resolved - at that point I was just sleeping through it.

I'll add more as we find out more, but it probably won't be till later this afternoon PST.

Edit: later than I expected, but for those still following, the main things I've learned are (1) pkill wasn't able to kill SBCL this time - we have a script that does that when HN stops responding, but it didn't work, so we'll revise the script; and (2) how to get PagerDuty not to let you go back to sleep if your site is actually still down.

>>dang+zk
I’d love to know more about what running a site like HN involves, would be great to get a write up of what it’s like running something like this at this scale (and what kind of traffic you guys get)!

>>andy_p+DC
I can’t put my finger on anything within the last decade, but I seem to recall it running in something close to its current form on a single core on a single server for a long time:

>>5229522

Re: traffic, dang said (2022):

>>33454140

I took it as a good reminder that the hard part is the human part: that high-overhead features and UI fripperies are nice but not necessary (or sufficient) to keep a community healthy and vibrant over the decades.

(And on the subject of the human side, if you didn’t catch Anna Wiener’s 2019 profile, it’s here:

https://www.newyorker.com/news/letter-from-silicon-valley/th... )