zlacker

Easier than that would just be downloading the torrent of all of Reddit through Sept 2023.

https://academictorrents.com/details/89d24ff9d5fbc1efcdaf9d7...

>>monoca+(OP)
That's fascinating that the total size is so tiny — only 2.4 TB‽

I assume this must be only the text portion, and heavily compressed?

>>q7xvh9+J
Text really doesn't take up that much space, and in addition it compresses pretty well.

The entire English language Wikipedia is only around 60GB in a format that can be readily searched and randomly accessed (ZIM), for example: https://kiwix.org/

>>lxgr+G5
Does Kiwix actually work? I see people hyping it here but I could never get it to actually, y'know, download the file and display the wikipedia on my phone.

>>lmm+C6
It works perfectly for me, both on iOS and macOS.

>>lmm+C6
Just downloaded. Doesn't seem to want to download Wikipedia on my phone, it says "detecting if filesystem supports 4gb files"

>>lmm+C6
Kiwix worked for me. IIRC there may be difficulties opening an archive that was downloaded outside of the mobile app, but archives downloaded in-app were fine.

For the mobile app I used one of the smaller Wikipedia subsets, since I didn't want to take up too much space on my phone. The full offline Wikipedia download is saved to my laptop.

>>monoca+(OP)
The question is: if you then added all of Usenet before, say, 1992, would the effective intelligence of the trained LLM go up or down?

>>PaulDa+ia
I can't speak to intelligence, but the result would be ignorant in a meaningful way.