zlacker

[parent] [thread] 0 comments
1. mschus+(OP)[view] [source] 2023-05-10 13:34:36
> Robots.txt has failed as a system, if it hadn't we wouldn't have captchas or Cloudflare.

That depends what you expect from it. For the purpose of limiting crawlers, at least the major search engines respect it.

[go to top]