zlacker

[return to "Tell HN: We should start to add “ai.txt” as we do for “robots.txt”"]
1. TechBr+Nr[view] [source] 2023-05-10 14:36:37
>>Jeanne+(OP)
If AI is using training data from your site, presumably it got that data by crawling it. So either it's already respecting robots.txt, in which case ai.txt would be redundant, or it's ignoring it, in which case there's no reason to expect it would respect ai.txt any more than it did robots.txt.
◧◩
2. asdkjl+vh2[view] [source] 2023-05-10 23:33:40
>>TechBr+Nr
exactly, but nobody in their sane minds would be using robots.txt to block crawlers as that is SEO (organic traffic from searches) disaster.

People could be adding a specific robot user-agent, if they knew openai even existed before yesterday and was stealing their content. but nobody did.

[go to top]