zlacker

[return to "Google to explore alternatives to robots.txt"]
1. Kwpols+ab[view] [source] 2023-07-08 07:51:20
>>skille+(OP)
Why would AI need a new standard for excluding it? Just add a "Googlebot-AI" user agent to your list [0] and respect these rules when crawling content for use in AIs, and convince OpenAI and Bing to do the same.

[0] https://developers.google.com/search/docs/crawling-indexing/...

◧◩
2. JohnFe+jW6[view] [source] 2023-07-10 16:01:04
>>Kwpols+ab
> and convince OpenAI and Bing to do the same.

And everyone else as well.

[go to top]