zlacker

[parent] [thread] 0 comments
1. kordle+(OP)[view] [source] 2023-05-10 13:34:36
robots.txt is for all crawlers, so there's no need for another file? robots.txt supports comments using # and ideally has a link to the site map, which would tell any robot crawler where the important bits live on the site.

Putting a good comment at the top of robots.txt would be just as good as any other solution, given it could serve as a type of prompt template for processing the data on the site it represents.

[go to top]