zlacker

I disagree it has failed as a system. While it does not substitute for authentication / authorization, reputable crawlers respect it, and there'd be a lot more traffic load on sites if they didn't have a way to tell reputable crawlers "please stop."

Similarly, extending robots.txt to direct AI would have a similar effect: not sufficient, but useful (if for no other reason than to make it easy to distinguish reputable AI projects from ones that feel like they own the Internet to do with as they please).