zlacker

[parent] [thread] 3 comments
1. retube+(OP)[view] [source] 2022-02-18 13:04:35
Thanks. Hoes does WBM get the page? Does the economist let it's crawlers through?
replies(3): >>jeroen+b >>Cthulh+57 >>flerch+S7
2. jeroen+b[view] [source] 2022-02-18 13:06:07
>>retube+(OP)
Crawlers are often permitted to get more web visibility. I think the Web Archive also crawls the web as a browser sometimes, so they can't always distinguish between crawlers.

That said, with NoScript on I can just read the economist article, no banners or anything.

3. Cthulh+57[view] [source] 2022-02-18 13:54:17
>>retube+(OP)
I do wonder; at some point, Google said they would penalize websites that show their crawlers something else than real visitors.
4. flerch+S7[view] [source] 2022-02-18 13:59:03
>>retube+(OP)
Just disable javascript and the entire article loads on the economist.
[go to top]