zlacker

I don’t think it’s necessarily saying the self-inflicted DDoS has caused a technical issue that’s forced them to shut down access. I think it’s possible that shutting down anonymous access caused the DDoS, which led to giant spikes in some metric, which led them (Elon) to conclude that there was an uptick in scraping, so they imposed the 600/tweet/day limit to punish scrapers.

Seems like either my quota reset or they changed the policy because I’m able to access the site again.

>>evan_+(OP)
This. I'd bet substantial amounts of money that the evil scraper idea is the result of a) another issue + b) paranoia + c) Musk thinking he understands better than anybody else.

>>pschue+7j
This is a really ignorant take to dismiss scrapers. LLMs operate by having petabytes of conversational training data. Scraping is how OpenAI trained GPT. It’s how all their copycats are trying to do the same.

Elon can be a monumental asshat, and he can be self-DDOS’ing, and can be accurate about scraping at the same time. It’s why every single social media platform is heading toward becoming a walled garden.

>>berkle+cp
I'm not denying that scrapers exist, I'm just highly suspicious of this explanation given that: a) he's proven time and time again how willing he is to say shit just to get attention b) he doesn't seem to understand software very well c) if shit was imploding for reasons related to decisions he made, this is precisely the kind of blame externalization I would expect.

>>pschue+5q
Yeah, scrapers have always existed and while their traffic is undoubtedly higher than it has been in the past, it can't possibly be any significant amount of traffic when compared to the rest of the traffic hitting the site.

A real scraper would be stopped by a rate limit set to, like, 100 tweets/minute. 600 tweets/day is a completely pointless, punitive limit.

>>berkle+cp
It’s quite ignorant to assume petabytes of garbage have any value at this point. See Chinchilla

>>evan_+(OP)
According to Elon, shutting down anonymous access was itself an emergency measure to deal with the DDoS[0]. Twitter did increase the quota significantly from earlier today[1].

[0] https://twitter.com/elonmusk/status/1674865731136020505

[1] https://twitter.com/elonmusk/status/1675260424109928449

>>olalon+Kv
The links won’t load

>>idiftl+uw
[0] "Temporary emergency measure. We were getting data pillaged so much that it was degrading service for normal users!" (in reference to Twitter requiring users to be logged in)

[1] "Now to 10k, 1k & 0.5k" (in reference to rate limits which were originally 6K 0.6K and 0.3K)

And another tweet that confirms disabling anonymous access was an emergency measure: https://twitter.com/elonmusk/status/1674942336583757825

> This will be unlocked shortly. Per my earlier post, drastic & immediate action was necessary due to EXTREME levels of data scraping.

> Almost every company doing AI, from startups to some of the biggest corporations on Earth, was scraping vast amounts of data.

> It is rather galling to have to bring large numbers of servers online on an emergency basis just to facilitate some AI startup’s outrageous valuation.

>>gmerc+zv
I agree, but there are hundreds if not thousands of AI startups trying to make their own relevant LLM, and they're going to be scraping Twitter. The Onion called it many years ago [1]: "400 billion tweets and not one useful bit of data was ever transmitted".

[1] https://www.youtube.com/watch?v=cqggW08BWO0&t=138s

>>berkle+qI
I can't imagine worse training data than e.g. Twitter and Reddit posts. How about like, dunno, books?

Edit: Ah, nvm, if you are trying to do a chat bot it is essentially what you want.

>>evan_+Oq
Depends on how they scrape .

100 tweets / minute is hardly deterrent for a botnet using comprised devices on non data center IPs

>>berkle+cp
How do you feel about search engine indexing?

>>evan_+Oq
> A real scraper would be stopped by a rate limit

I'm guessing you've never played an offensive or defensive role in scraping because what you've described is in no way a problem for a serious scraping effort. I agree the rate limits are stupid. They fuck over users, they stop amateur scrapers, and do nothing whatsoever to impede professional scraping.

If you want to stop most scraping, employ device attestation techniques and TLS fingerprinting.

>>berkle+jF1
But then you have to contend with this: https://github.com/bogdanfinn/tls-client... Just used this to bypass a Cloudflare check! I've never scraped Twitter but Elon said there was a large scraping operation from Oracle IPs. He could substantially raise the cost of scraping by just banning datacenter IPs. Something like p0f would probably help too. I pay for static residential proxies (basically servers running squid that somehow have IPs belonging to consumer ISPs) and with TCP fingerprinting these would be detected as Linux and expose my Windows or iPhone user-agents as inconsistent but I've never encountered a site that checks this. Although maybe sites are doing so silently but I don't notice because I don't otherwise meet the bot threshold.

>>costco+fV1
for sure, using a custom TLS library like uTLS helps -- need to inject that GREASE cipher selection. I have a suspicion that private residential proxies are out of budget for many outfits, or the IPs are too few and then simple rate limiting kicks in? Who do you use if you're willing to share? I've not been happy with the, uhh, questionable ethics of Luminati/BrightData in the past.

There are definitely more and more sites doing TLS/TCP/etc fingerprinting or device attestation for mobile APIs, but it's still pretty rare. I mean Twitter is trying to limit requests by IP, so definitely amateur hour over there.

>>berkle+I82
I use https://www.pingproxies.com/isp which is like $3/IP/month and unlimited bandwidth (I assume if you used a ridiculous amount they might charge you). Luminati pricing is extortionate. I have no idea how anyone doing anything at scale can afford $10/GB. I haven't investigated but I don't know if Twitter limits are per account or per IP.

>>costco+Ai2
Seriously. I don't even consider a provider if they want to charge for bandwidth. I'm doing about 50 TB/mo atm.