zlacker

>>skille+(OP)
Seems like it's intended for content stealing from every place that doesn't immediately implement Google's New Web Order as an addition to robots.txt.

"Your do not enter sign uses font we don't like, so we'll just ignore it"

>>voytec+V3
To be clear, robots.txt is not legally binding, Google is not bound to follow it, and in fact I believe that Google doesn't follow it and hasn't for a very long time, for the simple reason that many sites' robots.txt file is wrong.

The intent of robots.txt is to help crawlers, for example, to keep crawlers from getting stuck in a recursive loop of dynamic pages, or from crawling pages with no value. robots.txt is not for banning, restricting, or hindering crawlers.

>>Ferret+fh
for the simple reason that many sites' robots.txt file is wrong.

Which is of course not the real reason.

The reason Google doesn't follow the robots.txt protocol is (1) they don't want to (2) they can get away with it.