zlacker

[return to "Cloudflare outage on December 5, 2025"]
1. w10-1+aw[view] [source] 2025-12-05 17:47:25
>>meetpa+(OP)
Kudos to Cloudflare for clarity and diligence.

When talking of their earlier Lua code:

> we have never before applied a killswitch to a rule with an action of “execute”.

I was surprised that a rules-based system was not tested completely, perhaps because the Lua code is legacy relative to the newer Rust implementation?

It tracks what I've seen elsewhere: quality engineering can't keep up with the production engineering. It's just that I think of CloudFlare as an infrastructure place, where that shouldn't be true.

I had a manager who came from defense electronics in the 1980's. He said in that context, the quality engineering team was always in charge, and always more skilled. For him, software is backwards.

◧◩
2. zwnow+7G[view] [source] 2025-12-05 18:31:27
>>w10-1+aw
"Kudos"? This is like the South Park episode in which the oil company guy just excuses himself while the company just continues to fuck up over and over again. There's nothing to praise, this shouldn't happen twice in a month. Its inexcusable.
◧◩◪
3. vpShan+FU[view] [source] 2025-12-05 19:32:47
>>zwnow+7G
twice in a month _so far_
◧◩◪◨
4. hinkle+D61[view] [source] 2025-12-05 20:36:12
>>vpShan+FU
We still have two holidays and associated vacations and vacation brain to go. And then the January hangover.

Every company that has ignored my following advice has experienced a day for day slip in first quarter scheduling. And that advice is: not much work gets done between Dec 15 and Jan 15. You can rely on a week worth, more than that is optimistic. People are taking it easy and they need to verify things with someone who is on vacation so they are blocked. And when that person gets back, it’s two days until their vacation so it’s a crap shoot.

NB: there’s work happening on Jan 10, for certain, but it’s not getting finished until the 15th. People are often still cleaning up after bad decisions they made during the holidays and the subsequent hangover.

[go to top]