zlacker

To steelman this maybe, I think they’re angling for something like a mechanism to indicate content is OK to index but not OK to use as AI training data. Maybe you could fudge it today with user agents in robots.txt but who knows what the concrete idea of this is.

>>vore+(OP)
robots.txt is already outmoded. It only can indicate that content can’t be crawled but a URL marked this way can still be indexed. As Google says “it is not a mechanism for keeping a web page out of Google” [0] You need to use other things besides robots.txt to preventing indexing.

[0] https://developers.google.com/search/docs/crawling-indexing/...

>>varenc+i3
Indeed, having pages indexed which can't then be crawled is a great way of shooting yourself in the foot.

>>vore+(OP)
This seems weird to me though, aren't search engines something very similar to AI, if not outright AI?

>>dazc+26
I think you meant it's a great way for google to punish you for not giving them full access