zlacker

robots.txt is already outmoded. It only can indicate that content can’t be crawled but a URL marked this way can still be indexed. As Google says “it is not a mechanism for keeping a web page out of Google” [0] You need to use other things besides robots.txt to preventing indexing.

[0] https://developers.google.com/search/docs/crawling-indexing/...

>>varenc+(OP)
Indeed, having pages indexed which can't then be crawled is a great way of shooting yourself in the foot.

>>dazc+K2
I think you meant it's a great way for google to punish you for not giving them full access