zlacker

>>uyzstv+(OP)
Yes, sorry! We're investigating, but my current theory is we got overloaded because I relaxed some of our anti-crawler protections a few days ago.

(The reason I did that is that the anti-crawler protections also unfortunately hit some legit users, and we don't want to block legit users. However, it seems that I turned the knobs down too far.)

In this case, though, we had a secondary failure: PagerDuty woke me up at 5:24am, I checked HN and it seemed fine, so I told PagerDuty the problem was resolved. But the problem wasn't resolved - at that point I was just sleeping through it.

I'll add more as we find out more, but it probably won't be till later this afternoon PST.

Edit: later than I expected, but for those still following, the main things I've learned are (1) pkill wasn't able to kill SBCL this time - we have a script that does that when HN stops responding, but it didn't work, so we'll revise the script; and (2) how to get PagerDuty not to let you go back to sleep if your site is actually still down.

>>dang+zk
I didn't realize you were carrying the pager too! Kudos!

>>echelo+co
I feel such a sense of kinship for anyone who carries a pager, almost 7 years at my current role doing it. Super cool that dang is among our number :)

>>malwra+px
Do you carry a literal pager? We use the PagerDuty app.

>>idontw+OH
It may interest you to know that pagers are still a thing, Motorola still makes them, and I know that one major use case is volunteer fire departments

I used to work on Motorola Minitor 5 pagers. Looks like they recently released their newest model, the Minitor 7

I wonder if pagers are still used in hospitals? I imagine so

>>kunwon+oN
There's a company in England called "Cascode" who make firefighter alerters. These are really basic "beeper" pagers, which you can program to have a bunch of different tones and LED patterns based on the RIC and Subcode.

I look after several thousand of these across several hundred paging sites.

They're relatively inexpensive (70 quid or so in quantity) and they last about six weeks on a commonly-available AA battery. The batteries go flat enough to trigger the "low battery" beep at about 3am, for some reason. I don't know why.

There's no messaging involved, although the encoders are capable of sending a text string. The message is "get up and get down to the fire station right now", which generally needs no further explanation. POCSAG is unencrypted, so there would be privacy concerns with sending actual incident information in the clear with it.

While we're on the subject of old tech, until BT finally cut the last of them off, we use dialup modems to control the encoders (not dialup internet, just a hundreds-of-miles serial cable) as a backup, and dot-matrix printers to print out a hardcopy message for the crews to pick up.

All very low-tech. All very fixable. All stays working if you don't mess with it.

https://cascode.co.uk/products/2ar2-and-2ar3/