zlacker

I notice they don't actually give a good reason that robots.txt isn't suitable.

Change for the sake of it?

>>dbette+(OP)
AI!

>>dbette+(OP)
It doesn't require signing up with Google.

>>dbette+(OP)
To steelman this maybe, I think they’re angling for something like a mechanism to indicate content is OK to index but not OK to use as AI training data. Maybe you could fudge it today with user agents in robots.txt but who knows what the concrete idea of this is.

>>dbette+(OP)
> I notice they don't actually give a good reason that robots.txt isn't suitable

It's kind of implied: specifying sitemaps/allowance/copyright for different use cases: search, scraping, republishing, training etc. and perhaps adding some of the non standard extensions: Crawl-delay, default host, even sitemap isn't part of the robots.txt standard

> We believe it’s time for the web and AI communities to explore additional machine-readable means for web publisher choice and control for emerging AI and research use cases.

>>stromb+5
Indeed, AI is called out directly. Google’s laying groundwork for their own version of a regulatory moat.

>>vore+t
robots.txt is already outmoded. It only can indicate that content can’t be crawled but a URL marked this way can still be indexed. As Google says “it is not a mechanism for keeping a web page out of Google” [0] You need to use other things besides robots.txt to preventing indexing.

[0] https://developers.google.com/search/docs/crawling-indexing/...

>>Animat+7
If it would be public the ai could read it and can develop countermeasures ;) .

>>varenc+L3
Indeed, having pages indexed which can't then be crawled is a great way of shooting yourself in the foot.

>>vore+t
This seems weird to me though, aren't search engines something very similar to AI, if not outright AI?

>>dazc+v6
I think you meant it's a great way for google to punish you for not giving them full access

>>dbette+(OP)
Came here to say that, seems like nobody as the answer :/

Maybe they want to have finer details on page content, e.g: "you can index those pages but not those nodes" or "those nodes are also AI generated please ignore".

Otherwise I don't know, robots.txt is not sexy but definitely does the job.

>>dbette+(OP)
I think robots.txt isn't suitable for this for the same reason it's not suitable for keeping other bots from crawling your site: adhering to what robots.txt says is optional, and plenty of bots opt to ignore it.