zlacker

[return to "Cloudflare outage on December 5, 2025"]
1. uyzstv+6m[view] [source] 2025-12-05 17:03:50
>>meetpa+(OP)
What I'm missing here is a test environment. Gradual or not; why are they deploying straight to prod? At Cloudflare's scale, there should be a dedicated room in Cloudflare HQ with a full isolated model-scale deployment of their entire system. All changes should go there first, with tests run for every possible scenario.

Only after that do you use gradual deployment, with a big red oopsie button which immediately rolls the changes back. Languages with strong type systems won't save you, good procedure will.

◧◩
2. eviks+8n2[view] [source] 2025-12-06 08:57:47
>>uyzstv+6m
> Languages with strong type systems won't save you, good procedure will.

One of the items in the list of procedures is to use types to encode rules of your system.

[go to top]