zlacker

[return to "Tell HN: HN was down"]
1. dang+zk[view] [source] 2025-12-17 18:09:25
>>uyzstv+(OP)
Yes, sorry! We're investigating, but my current theory is we got overloaded because I relaxed some of our anti-crawler protections a few days ago.

(The reason I did that is that the anti-crawler protections also unfortunately hit some legit users, and we don't want to block legit users. However, it seems that I turned the knobs down too far.)

In this case, though, we had a secondary failure: PagerDuty woke me up at 5:24am, I checked HN and it seemed fine, so I told PagerDuty the problem was resolved. But the problem wasn't resolved - at that point I was just sleeping through it.

I'll add more as we find out more, but it probably won't be till later this afternoon PST.

Edit: later than I expected, but for those still following, the main things I've learned are (1) pkill wasn't able to kill SBCL this time - we have a script that does that when HN stops responding, but it didn't work, so we'll revise the script; and (2) how to get PagerDuty not to let you go back to sleep if your site is actually still down.

◧◩
2. rldjbp+Zj2[view] [source] 2025-12-18 10:07:27
>>dang+zk
> The reason I did that is that the anti-crawler protections also unfortunately hit some legit users, and we don't want to block legit users.

it is a shame that it needs to be this way. as a lurker who doesn't stay logged in nor use incognito mode, i have seen "Sorry" page way too often, even when opening the "past" page from the homepage.

truly hope you find a solution that reduces friction for all. personally, it is back to "Sorry" situation for now.

PS: for others facing a similar situation, it all disappears after logging in, which has been the most reliable solution thus far.

[go to top]