zlacker

Robots.txt is meant as an aid to crawlers. "This stuff is not useful to index," rather than a blocking mechanism