zlacker

[return to "Cloudflare outage on December 5, 2025"]
1. uyzstv+6m[view] [source] 2025-12-05 17:03:50
>>meetpa+(OP)
What I'm missing here is a test environment. Gradual or not; why are they deploying straight to prod? At Cloudflare's scale, there should be a dedicated room in Cloudflare HQ with a full isolated model-scale deployment of their entire system. All changes should go there first, with tests run for every possible scenario.

Only after that do you use gradual deployment, with a big red oopsie button which immediately rolls the changes back. Languages with strong type systems won't save you, good procedure will.

◧◩
2. vouwfi+OW[view] [source] 2025-12-05 19:43:56
>>uyzstv+6m
> Languages with strong type systems won't save you

Neither will seatbelts if you drive into the ocean, or helmets if you drink poison. I'm not sure what your point is.

◧◩◪
3. djmips+2f1[view] [source] 2025-12-05 21:16:56
>>vouwfi+OW
I think you strengthened their point.
[go to top]