zlacker

This is a really ignorant take to dismiss scrapers. LLMs operate by having petabytes of conversational training data. Scraping is how OpenAI trained GPT. It’s how all their copycats are trying to do the same.

Elon can be a monumental asshat, and he can be self-DDOS’ing, and can be accurate about scraping at the same time. It’s why every single social media platform is heading toward becoming a walled garden.

>>berkle+(OP)
I'm not denying that scrapers exist, I'm just highly suspicious of this explanation given that: a) he's proven time and time again how willing he is to say shit just to get attention b) he doesn't seem to understand software very well c) if shit was imploding for reasons related to decisions he made, this is precisely the kind of blame externalization I would expect.

>>pschue+T
Yeah, scrapers have always existed and while their traffic is undoubtedly higher than it has been in the past, it can't possibly be any significant amount of traffic when compared to the rest of the traffic hitting the site.

A real scraper would be stopped by a rate limit set to, like, 100 tweets/minute. 600 tweets/day is a completely pointless, punitive limit.

>>berkle+(OP)
It’s quite ignorant to assume petabytes of garbage have any value at this point. See Chinchilla

>>gmerc+n6
I agree, but there are hundreds if not thousands of AI startups trying to make their own relevant LLM, and they're going to be scraping Twitter. The Onion called it many years ago [1]: "400 billion tweets and not one useful bit of data was ever transmitted".

[1] https://www.youtube.com/watch?v=cqggW08BWO0&t=138s

>>berkle+ej
I can't imagine worse training data than e.g. Twitter and Reddit posts. How about like, dunno, books?

Edit: Ah, nvm, if you are trying to do a chat bot it is essentially what you want.

>>evan_+C1
Depends on how they scrape .

100 tweets / minute is hardly deterrent for a botnet using comprised devices on non data center IPs

>>berkle+(OP)
How do you feel about search engine indexing?

>>evan_+C1
> A real scraper would be stopped by a rate limit

I'm guessing you've never played an offensive or defensive role in scraping because what you've described is in no way a problem for a serious scraping effort. I agree the rate limits are stupid. They fuck over users, they stop amateur scrapers, and do nothing whatsoever to impede professional scraping.

If you want to stop most scraping, employ device attestation techniques and TLS fingerprinting.

>>berkle+7g1
But then you have to contend with this: https://github.com/bogdanfinn/tls-client... Just used this to bypass a Cloudflare check! I've never scraped Twitter but Elon said there was a large scraping operation from Oracle IPs. He could substantially raise the cost of scraping by just banning datacenter IPs. Something like p0f would probably help too. I pay for static residential proxies (basically servers running squid that somehow have IPs belonging to consumer ISPs) and with TCP fingerprinting these would be detected as Linux and expose my Windows or iPhone user-agents as inconsistent but I've never encountered a site that checks this. Although maybe sites are doing so silently but I don't notice because I don't otherwise meet the bot threshold.

>>costco+3w1
for sure, using a custom TLS library like uTLS helps -- need to inject that GREASE cipher selection. I have a suspicion that private residential proxies are out of budget for many outfits, or the IPs are too few and then simple rate limiting kicks in? Who do you use if you're willing to share? I've not been happy with the, uhh, questionable ethics of Luminati/BrightData in the past.

There are definitely more and more sites doing TLS/TCP/etc fingerprinting or device attestation for mobile APIs, but it's still pretty rare. I mean Twitter is trying to limit requests by IP, so definitely amateur hour over there.

>>berkle+wJ1
I use https://www.pingproxies.com/isp which is like $3/IP/month and unlimited bandwidth (I assume if you used a ridiculous amount they might charge you). Luminati pricing is extortionate. I have no idea how anyone doing anything at scale can afford $10/GB. I haven't investigated but I don't know if Twitter limits are per account or per IP.

>>costco+oT1
Seriously. I don't even consider a provider if they want to charge for bandwidth. I'm doing about 50 TB/mo atm.