zlacker

[return to "Cloudflare outage on December 5, 2025"]
1. w10-1+aw[view] [source] 2025-12-05 17:47:25
>>meetpa+(OP)
Kudos to Cloudflare for clarity and diligence.

When talking of their earlier Lua code:

> we have never before applied a killswitch to a rule with an action of “execute”.

I was surprised that a rules-based system was not tested completely, perhaps because the Lua code is legacy relative to the newer Rust implementation?

It tracks what I've seen elsewhere: quality engineering can't keep up with the production engineering. It's just that I think of CloudFlare as an infrastructure place, where that shouldn't be true.

I had a manager who came from defense electronics in the 1980's. He said in that context, the quality engineering team was always in charge, and always more skilled. For him, software is backwards.

◧◩
2. braiam+qD[view] [source] 2025-12-05 18:20:24
>>w10-1+aw
This is funny, considering that someone that worked on the defense industry (guide missile system) found a memory leak on one of their products, at that time. They told him that they knew about it, but that it's timed just right with the range of the system it would be used, so it doesn't matter.
◧◩◪
3. mopsi+9K[view] [source] 2025-12-05 18:49:22
>>braiam+qD
... until the extended-range version is ordered and no one remembers to fix the leak. :]
◧◩◪◨
4. wizzwi+TU[view] [source] 2025-12-05 19:33:49
>>mopsi+9K
They will remember, because it'll have been measured and documented, rigorously.
◧◩◪◨⬒
5. Sketch+IW[view] [source] 2025-12-05 19:43:40
>>wizzwi+TU
I've found that the real trick with documentation isn't creation, it's discovery. I wonder how that information is easily found afterwards.
◧◩◪◨⬒⬓
6. lloeki+dY[view] [source] 2025-12-05 19:50:59
>>Sketch+IW
By reading the documentation thoroughly as a compulsory first step to designing the next system that depends on it.

I realise this may probably boggle the mind of the modern software developer.

◧◩◪◨⬒⬓⬔
7. lukan+Y31[view] [source] 2025-12-05 20:21:49
>>lloeki+dY
You say this like trivial misstakes did not happen all the time in classical engineering as well.

If there is a memory leak, them this is a flaw, that might not matter so much for a specific product, but I can also easily see it being forgotten, if it was maybe mentioned somewhere in the documentation, but maybe not clear enough and deadlines and stress to ship are a thing there as well.

[go to top]