zlacker

>>skille+(OP)
Why would AI need a new standard for excluding it? Just add a "Googlebot-AI" user agent to your list [0] and respect these rules when crawling content for use in AIs, and convince OpenAI and Bing to do the same.

[0] https://developers.google.com/search/docs/crawling-indexing/...

>>Kwpols+ab
Google just copies everything and saves it. They then use it for various uses. It would be strange to fetch pages several times.

>>lokar+MR
They don't really need to fetch it twice though? Fetch 0-1 times, use it according to what robots.txt allows.