zlacker

[parent] [thread] 0 comments
1. juliel+(OP)[view] [source] 2023-05-11 08:33:26
It is fair to give more information about the information exposed on a website, especially when it comes to partnering with AI systems. There is an international effort which includes such information. It is done under the auspices of the W3C. See https://www.w3.org/community/tdmrep/. It has been developed to implement the Text & Data Mining + AI "opt-out" that is legal in Europe. It does not use robots.txt because this one is about indexing a website and should stay focus on it. The information about website managers is contained in the /.well-known directory, in a JSON-LD file, which is much more well structured than robots.txt. Why not adhere to an international effort rather than creating N fragmented initiatives?
[go to top]