zlacker

If you want training data for an LLM and are actively talking to some data providers, you'd just ask for a dump, instead of making a billion small requests.

(You'd make the billion small requests, if you are doing this on the sly.)

>>eru+(OP)
Right that'd be the case now but previously you could just make a billion small requests for free.

>>sahila+Sj
Or at least you could try.

But that still makes the original commenters argument moot:

> Enterprise deals between these user generated content platforms and LLM platforms may well involve many billions of API requests, and the pricing is likely an order of magnitude less expensive per call due to the volume. The result is a cost-per-call that is cost-prohibitive at smaller scales, [...]

That speculation is not how things have been or were.

>>eru+Sw
I think most people who wanted large datasets got their data via pushshift. Pushshift was basically a guy who started out doing small things got so frustrated with the API that he eventually grew to maintaining large mirrors of Reddit content on Google cloud that people could access and query. I don't know why anyone doing research would have used reddit's API instead of using pushshift.

Pushshift has been shutdown by reddit earlier this year, so probably they are getting hammered by LLM folks trying to get the data now since they killed pushshift without understanding how it fit into the universe.

Reddit is completely stupid if they think people are going to pay for "enterprise API" access... pushshift existed because the API was trash and the only real option is to dump the entire dataset into something usable. The reason reddit's data was used so much is because there was an SQL API via pushshift and you could also download archives of the entire dataset at one go.

>>fluidc+pJ
> Pushshift has been shutdown by reddit earlier this year

Oh is this why all the comment undelete websites broke?

>>eru+(OP)
Eh can still just automate creating a bunch of accounts and do it manually. Use one of the many captcha completion services where you pay for people to complete captchas for you. ML models can already pretty much do them anyway.

Then rotate between accounts and put a random time between requests. Restrict certain accounts to browse within certain hours/timezones. Load pages as usual and just scrape data from the page rather than via api.

However, I believe in a company's right to charge whatever they want for their services. But I also believe in the right for people to choose not to use that service and for freer alternatives to spring up.

Just like Tumblr, Reddit seem intent on killing themselves, although these days I'm not so sure. When Elon took over Twitter everyone was saying that all the users would leave and it would die. This is not the case, human nature means that people seek familiarity and will cling on, hmm.

>>doglea+x71
Yep this is exactly why

>>eru+(OP)
You’re right, but I think it’s also pretty clear that

A) there is demand for functionality that depends on semi-real-time data, e.g. a prompt like “explain {recent_trending_topic} to me and describe its evolution” where the return could be useful in various contexts;

B) the degradation of search experience and the explosion of chat interfaces seem to indicate “the future of search is chat” and the number of Google searches prefixed or suffixed with “Reddit” make it obvious that LLM-powered chat models with search functionality will want to query Reddit extensively, and in the example prompt above, the tree of queries generated to fulfill a single prompt could be sizeable;

C) improvements to fine-tuning pipelines make it more and more feasible to use real-time data in the context of LLMs, such as a “trending summary” function that could cache many potentially related queries from Reddit, Twitter, etc and use them to fine-tune a model which would serve a response to my example prompt

>>fennec+0t1
> Just like Tumblr, Reddit seem intent on killing themselves, although these days I'm not so sure. When Elon took over Twitter everyone was saying that all the users would leave and it would die. This is not the case, human nature means that people seek familiarity and will cling on, hmm.

Wouldn't network effects be the obvious null hypothesis, before we start speculating about human nature?