zlacker

>>skille+(OP)
Seems like it's intended for content stealing from every place that doesn't immediately implement Google's New Web Order as an addition to robots.txt.

"Your do not enter sign uses font we don't like, so we'll just ignore it"

>>voytec+V3
To be clear, robots.txt is not legally binding, Google is not bound to follow it, and in fact I believe that Google doesn't follow it and hasn't for a very long time, for the simple reason that many sites' robots.txt file is wrong.

The intent of robots.txt is to help crawlers, for example, to keep crawlers from getting stuck in a recursive loop of dynamic pages, or from crawling pages with no value. robots.txt is not for banning, restricting, or hindering crawlers.

>>Ferret+fh
They are in the EU. If something was not meant to be accessible you may not scrape it.