zlacker

Large scale infrastructure changes are often by nature completely untestable. The system is too large, there are too many moving parts to replicate with any kind of sane testing, so often, you do find out in prod, which is why robust and fast rollback procedures are usually desirable and implemented.

>>JohnMa+(OP)
> Large scale infrastructure changes are often by nature completely untestable.

You're changing the subject here and shifting focus from the specific to the vague. The two postmortems after the recent major Cloudflare outages both listed straightforward errors in source code that could have been tested and detected.

Theoretical outages could theoretically have other causes, but these two specific outages had specific causes that we know.

> which is why robust and fast rollback procedures are usually desirable and implemented.

Yes, nobody is arguing against that. It's a red herring with regard to my point about source code testing.

>>lapcat+M3
I am not changing any subject. These are glue logic scripts connecting massive pieces of infra together, spanning what is likely several teams and orgs over the course of many years. It is impossible to blurt something out like "well, source code testing" for something like this, when the source code inputs are not possibly testable outside the scale of the larger system. They're often completely unknowable as well.

With all due respect, it sounds like you have not worked on these types of systems, but out of curiosity - what type of test do you think would have prevented this?

>>JohnMa+oi
With all due respect, it sounds like you have never heard of unit tests.

Cloudflare states that the compiler would prevent the bug in certain programming languages. So it seems ridiculous to suggest that the bug can't be detected outside the scale of a larger system.

>>JohnMa+(OP)
Akamai manages it.

>>roguec+Xu
They don't, akamai has had several outages as well jsut no one notices. Akamai is way way smaller than cloudflare, 20% of internet traffic passes through CF networks, not sure it's even measurable on Akamai.

>>winddu+iV
Quickly Googling about, a commonly repeated figure is that Akamai served 15% - 30% of Internet traffic in the late 2010's. They probably have less of the market today due to others growing, but they're not a minnow.

2024 revenue figures were $1.669 billion for Cloudflare, and $3.99 billion for Akamai, per Wikipedia.

>>lapcat+Jl
Please explain how unit tests stop a problem from propagating across a system that fields 70 million requests a second and I’ll take you more seriously, otherwise I’m done with this particular subthread.

>>JohnMa+bl1
You're completely missing the point. From the blog post:

  if rule_result.action == "execute" then
    rule_result.execute.results = ruleset_results[tonumber(rule_result.execute.results_index)]
  end

"This code expects that, if the ruleset has action=”execute”, the “rule_result.execute” object will exist. However, because the rule had been skipped, the rule_result.execute object did not exist, and Lua returned an error due to attempting to look up a value in a nil value.

This is a straightforward error in the code, which had existed undetected for many years. This type of code error is prevented by languages with strong type systems. In our replacement for this code in our new FL2 proxy, which is written in Rust, the error did not occur."

The unit tests are for the source code. In this respect, the number of requests a second fielded by the system is irrelevant. Unit tests don't happen in production; that's the point of them.

It's a classic coding mistake, failing to check for nil, and none of your handwaving about "scale" changes that fact.

>>andrew+uc1
https://w3techs.com/technologies/overview/proxy, they are tiny compared to CF, their revenue is high because they focus on large enterprise clients.

>>winddu+5q1
> 20% of internet traffic passes through CF networks

That does not sound right to me. “20 percent of websites” does not mean “20 percent of traffic.”.

There is no public write-up from Cloudflare that proves “we handle 20% of all Internet traffic.” Cloudflare reports around 295,000 paying customers and more than 30 million Internet properties (20% of the web). So most of their users are on the free plan.

>>z00z1+Fx2
agreed, cloudflare says 20% of all websties, which makes sense, digging into it the "20% of all web traffic" seems to have been like a game of telephone in media, social media, and/or AI.

I can't believe they only have 295,000 paying customer, that puts me in a small minority. lol