zlacker
[return to "Google to explore alternatives to robots.txt"]
◧
1. Kwpols+ab
[view]
[source]
2023-07-08 07:51:20
>>skille+(OP)
Why would AI need a new standard for excluding it? Just add a "Googlebot-AI" user agent to your list [0] and respect these rules when crawling content for use in AIs, and convince OpenAI and Bing to do the same.
[0]
https://developers.google.com/search/docs/crawling-indexing/...
◧◩
2. JohnFe+jW6
[view]
[source]
2023-07-10 16:01:04
>>Kwpols+ab
> and convince OpenAI and Bing to do the same.
And everyone else as well.
[go to top]