(The reason I did that is that the anti-crawler protections also unfortunately hit some legit users, and we don't want to block legit users. However, it seems that I turned the knobs down too far.)
In this case, though, we had a secondary failure: PagerDuty woke me up at 5:24am, I checked HN and it seemed fine, so I told PagerDuty the problem was resolved. But the problem wasn't resolved - at that point I was just sleeping through it.
I'll add more as we find out more, but it probably won't be till later this afternoon PST.
Edit: later than I expected, but for those still following, the main things I've learned are (1) pkill wasn't able to kill SBCL this time - we have a script that does that when HN stops responding, but it didn't work, so we'll revise the script; and (2) how to get PagerDuty not to let you go back to sleep if your site is actually still down.
Was the blocking returning “Sorry.” instead of any page content? A couple of days ago there was a few hours where when I’d go to HN I could load the main page as a non-logged in user. But if I tried to log in I would get “Sorry.” instead. I also got the sorry message if I tried to click on user profiles of other people and a few other pages.
I am assuming that the reason I could see the front page itself and discussions on posts on the front page is that they were in a shared cache for non-logged in users, but that when I clicked on some pages like some random user pages those were not in cache and hit the origin server and it blocked those with “Sorry.” like it did for log-in attempts.
I also tried to go to the unblock IP page, but that one also returned “Sorry.”
For a while I was scratching my head wondering if I had gotten some malware on one of my computers that was aggressively making requests to HN, and that I had become IP banned because of that. Since I think my actual request rate from browsing and commenting should be pretty average. I read HN a lot, but not that much :p
Later in the day, or the next day, things were back to normal and I could log in again. Presumably after those anti-crawler protections had been relaxed again.
> Presumably after those anti-crawler protections had been relaxed again
Yup and yup. Apologies for the inconvenience! If it happens again you're welcome to email us at hn@ycombinator.com with your IP and we'll unblock it for you.