zlacker

This is interesting.

Judging from the screenshot, a huge amount of GET /TweetDetail is generated which triggers some rate limiting, as shown by the 429.

If this is indeed due to the recent decision to enforce authentication for all API calls, it means the curlprit may actually be the API gateway or something similar downstream.

Also, this behavior seem to never stop, which isn't what one would expect from an exponential backoff retry.

I don't claim to be a better engineer than the folks working at Twitter, but it is interesting to see something like this in the wild, all Musk-related considerations aside.

>>arter4+(OP)
> If this is indeed due to the recent decision to enforce authentication for all API calls, it means the curlprit may actually be the API gateway or something similar downstream.

The way I understand it, DDoS is not caused by enforced authentication - enforced authentication is just a temporary measure against DDoS.

>>arter4+(OP)
I would guess the front end was written under the assumption that the back end would still work without auth. Perhaps the backend changes (mandatory auth + rate limiting) were pushed without sufficient testing of the front + back?

>>arter4+(OP)
Did Elon pay the AWS bill? That seems like a likely culprit. Twitter instances are being forcibly shutdown.

>>cactus+2h
Twitter operates its own datacenters.

>>amluto+1i
"Twitter and AWS signed a five-and-a-half-year contract in 2020, which AWS is not willing to renegotiate."

https://gritdaily.com/twitter-owes-aws-millions/

>>cactus+Ui
Twitter.com and the associated user-facing services do not run on AWS.

>>arter4+(OP)
"curlprit" for too many GET's causing a 429 is just the perfect typo.

>>amluto+1i
Yeah but they use GCS for auth, moderation, and caching. They apparently haven’t been paying Google since April and the contract expired June 30th

>>arter4+(OP)
Could someone report the error at press@twitter.com and see what they think about it?

>>cactus+2h
Well, they haven't paid their GCP bill... https://theconversation.com/twitter-is-refusing-to-pay-googl...

>>amluto+1i
And yet they also host with AWS, Google Cloud and Oracle. Cloud people take note: this is what lock-in looks like, and it's coming to a place near you.

>>stefan+LA
No, this has nothing to do with lock-in. This has done nothing to do with decision making that subverts good engineering.

>>readyp+ap
Quite the knee slapper. Thanks. I hope you didn’t spend many brain cycles on it.

>>willia+Vj
While it looks like they never started the move over to AWS, the press release makes it sound like they do use some AWS services.

> In addition, Twitter will continue to use AWS services such as Amazon CloudFront (AWS’s fast content delivery network service that securely delivers data, videos, applications, and APIs with low latency and high transfer speeds to customers globally) and Amazon DynamoDB (AWS’s key-value database that delivers single-digit millisecond performance at any scale).

>>vGPU+wG
I worked there. Services running on GCP are a significant part of the internal service infra (ml platform, etc.) and it's not impossible that the abrupt loss of GCP would cause user-facing problems. The GCP spend was many, many times the AWS spend. Unless things changed since last November, AWS is not a meaningful part of the internal or user-facing infra.

With respect to DynamoDB specifically, Twitter has its own custom distributed key-value store: https://blog.twitter.com/engineering/en_us/a/2014/manhattan-... that twitter.com itself runs on.

>>stefan+LA
Cloud agnostic is hard

>>arter4+(OP)
While exponential backoff is theoretically optimal, I doubt it's actually used that often in practice. I've seen too many cases where someone decides serving user requests with low latency is so important that they'd rather have a constant randomized backoff than exponential backoff. I've been in many design meetings and seen enough documents where the decision not to use exponential backoff is explicitly made, understanding the tradeoff with overloading and system recovery.

>>vuln+MF
Making jokes about Twitter is too easy.

>>willia+sH
Thanks for weighing in with some actual first-hand knowledge. It is appreciated.

The latest on cloud hosting is from a week ago, and I'm guessing you don't have any more recent info than this:

https://www.reuters.com/technology/twitter-resumes-paying-go...

>>18pfsm+KL
Correct, no more recent (or less public) info than that. Like I say, losing GCP could cause problems users notice, but sounds like that’s not going to happen.

>>kccqzy+VK
I wouldn’t be surprised if this has no back off or limit at all. 10 RPS is fast enough that it may simply be sequential.

>>readyp+ap
Whoever is in charge of that account went all Oregon Trail on things and caught dysentery

>>vuln+MF
Why do you take people criticizing Elon Musk so personally?

>>vuln+MF
Do you think it's more or less funny than auto-replying to all PR enquiries with poo, when you are an incredibly large company?

>>badwol+Jv
That's slightly outdated information:

https://www.engadget.com/twitter-has-supposedly-started-payi...

>>kccqzy+VK
I’ve had to… uhh… eagerly advocate for exponential backoff for weeks of constant uptime issues before someone listened and actually implemented it and solved the problems.

Like several times in different roles.

People do it, exponential backoff is everywhere in your stack, but it doesn’t end up in your application layer until you have enough traffic that you actually have to manage throughout.

>>cactus+2h
This is not remotely a likely culprit.