(The reason I did that is that the anti-crawler protections also unfortunately hit some legit users, and we don't want to block legit users. However, it seems that I turned the knobs down too far.)
In this case, though, we had a secondary failure: PagerDuty woke me up at 5:24am, I checked HN and it seemed fine, so I told PagerDuty the problem was resolved. But the problem wasn't resolved - at that point I was just sleeping through it.
I'll add more as we find out more, but it probably won't be till later this afternoon PST.
Edit: later than I expected, but for those still following, the main things I've learned are (1) pkill wasn't able to kill SBCL this time - we have a script that does that when HN stops responding, but it didn't work, so we'll revise the script; and (2) how to get PagerDuty not to let you go back to sleep if your site is actually still down.
I used to work on Motorola Minitor 5 pagers. Looks like they recently released their newest model, the Minitor 7
I wonder if pagers are still used in hospitals? I imagine so
I look after several thousand of these across several hundred paging sites.
They're relatively inexpensive (70 quid or so in quantity) and they last about six weeks on a commonly-available AA battery. The batteries go flat enough to trigger the "low battery" beep at about 3am, for some reason. I don't know why.
There's no messaging involved, although the encoders are capable of sending a text string. The message is "get up and get down to the fire station right now", which generally needs no further explanation. POCSAG is unencrypted, so there would be privacy concerns with sending actual incident information in the clear with it.
While we're on the subject of old tech, until BT finally cut the last of them off, we use dialup modems to control the encoders (not dialup internet, just a hundreds-of-miles serial cable) as a backup, and dot-matrix printers to print out a hardcopy message for the crews to pick up.
All very low-tech. All very fixable. All stays working if you don't mess with it.
You wouldn't even need particularly good encryption, you'd just need something adequate to stop casual eavesdropping really - "keep them busy for half an hour" would stop people from sniffing the POCSAG traffic and tweeting it, so that people show up at incidents and hang around filming it on their phones.
This incidentally is what a guy in England got arrested for a few years ago, exactly that. It's perfectly legal to listen to and decode pager messages (or any other radio messages), you're just not allowed to pass them on to people or act upon them, and posting them on twitter and then going round to rubberneck at the ongoing incident very much ticks those boxes. As with so many things in the UK, to paraphrase Aleister Crowley, "Don't Be A Dick shall be the whole of the law".