I have seen similar bugs in the systems I oversee because network libraries love to retry requests without sane limitations by default. But I never saw them make our rate limiters sweat. It's slightly more annoying when they hit an API which actually does some expensive work before returning an error but that's why we have rate limits on all public endpoints.
I also guess that the webapp is the least of Twitters traffic and the native apps probably don't have this problem.
I agree that this is probably not the bug at the root of it all. But I also don't believe the story that Musk is selling for why he's in effect shutting down the site. But both could be true and I'm still thinking about other potential reasons, a complete waste of my time, but it's a weird mental honeypot.
The book "Nothing is true and everything is possible" describes Putin's use of misinformation to maintain control of the populace and eliminate democratic types of politics, but it really feels like it applies here too. There will always be Musk fanbois who will parrot whatever he wants them to say, but most know it's just self-serving BS. And anybody trying to get to the root of everything gets easily sidetracked into narratives that feel right but have zero data backing them, like this bug.
Anyway, highly recommend this book if you want to see a likely path for the future of the US:
https://en.wikipedia.org/wiki/Nothing_Is_True_and_Everything...
I will note that the few times I investigated claims of Elon lies they were not proper lies, either being misunderstood, misleading (which IS unethical, don't get me wrong), of indeterminate truth value (he said, she said type stuff), delusional optimism or actually true.
Like journalists, Musk rarely outright knowingly makes literally false statements, but this does not mean you should take what he says at face value.
Seems like either my quota reset or they changed the policy because I’m able to access the site again.
"Nothing is an absolute reality, all is permitted"
An assumptions of good faith was once a well-held principle on this website, and it's too bad legacy media has led so many astray.
Hatred for Musk has truly captured many otherwise very logical minds.
Infact it got so bad because of all those retries at multiple levels from upstream callers that requests were essentially timing out at the TCP buffer/queue before they could be processed by the application.
Don’t know if the Twitter homepage backend is at similar scale.
I prefer to assume the best out of people, but when someone is that obviously wrong that many times with that much personal gain to come from it, I can't believe that man as smart as he is would also be that misled.
Is it a lie if you don't know or even care if it's true or what it means? That's where we are with Elon Musk.
My favorite is when things are claimed to be impossible, that Elon's lying because the thing is impossible, but then it happens anyway.
Where Elon gets in trouble is he's wildly over-optimistic on a few things, such as AI. He predicted an AGI would take over in 5 years around 2015 or so (so we're 3 years off), and I think he really believed it. That's why he's always claiming things like self-driving, and doing it without sensors or whatever. His paranoia of AGI and his over-confidence on self-driving have exactly the same root cause (believing AI will conquer all). Elon has had so many instances of overcoming status quo expert predictions (whether on solar energy, battery-electric vehicles, reusable launch vehicles, or whathaveyou), that I think it makes him increasingly unable to very effectively listen to experts.
And he's also incredibly gullible and easily taken in by all sorts of scammers, including rightists and just plain sycophants telling him what they think he wants to hear. Which is increasingly what he's left with as everyone else who is sick of his bull has left.
The media is flawed. Journalists are flawed. The Verge in particular has some atrocious coverage of tech.
But the problem with Elon is that in his world, we must rely on him for true information. As if the direct source is inherently true. He’s just as full of it as anyone else with interests.
Elon can be a monumental asshat, and he can be self-DDOS’ing, and can be accurate about scraping at the same time. It’s why every single social media platform is heading toward becoming a walled garden.
A real scraper would be stopped by a rate limit set to, like, 100 tweets/minute. 600 tweets/day is a completely pointless, punitive limit.
If IPs or IP ranges get really annoying we block them on the network level.
Big public sites like Twitter obviously need to have this technology. Due to their political content they probably also need sophisticated DDoS protection.
[1] "Now to 10k, 1k & 0.5k" (in reference to rate limits which were originally 6K 0.6K and 0.3K)
And another tweet that confirms disabling anonymous access was an emergency measure: https://twitter.com/elonmusk/status/1674942336583757825
> This will be unlocked shortly. Per my earlier post, drastic & immediate action was necessary due to EXTREME levels of data scraping.
> Almost every company doing AI, from startups to some of the biggest corporations on Earth, was scraping vast amounts of data.
> It is rather galling to have to bring large numbers of servers online on an emergency basis just to facilitate some AI startup’s outrageous valuation.
Or Google "Elon Musk conspiracy" and click the top link. I don't know if you'll be able to read the tweets they reference though.
Edit: Ah, nvm, if you are trying to do a chat bot it is essentially what you want.
He should have lost that libel case imo as I do think he meant it.
100 tweets / minute is hardly deterrent for a botnet using comprised devices on non data center IPs
I'm guessing you've never played an offensive or defensive role in scraping because what you've described is in no way a problem for a serious scraping effort. I agree the rate limits are stupid. They fuck over users, they stop amateur scrapers, and do nothing whatsoever to impede professional scraping.
If you want to stop most scraping, employ device attestation techniques and TLS fingerprinting.
There are definitely more and more sites doing TLS/TCP/etc fingerprinting or device attestation for mobile APIs, but it's still pretty rare. I mean Twitter is trying to limit requests by IP, so definitely amateur hour over there.
Saying "he really believes his lies" is not excuse, because most habitual liars are exactly the same. They have prioritized their own narcissism so far above reality, but that doesn't make them any less of a liar. They are just lying to themselves and everyone else.