zlacker

[parent] [thread] 1 comments
1. deadal+(OP)[view] [source] 2021-09-11 21:00:18
archive.org obeys the robots.txt exclusion but archive.today doesn't. This means that many websites(like 4chan) cannot be archived with archive.org.
replies(1): >>oeneta+I1
◧
2. oeneta+I1[view] [source] 2021-09-11 21:08:03
>>deadal+(OP)
That's true, so does webcitation.org, which doesn't obey robots.txt.

4chan has archives dedicated for that site anyway (Warosu, etc.).

[go to top]