zlacker

>>skille+(OP)
I notice they don't actually give a good reason that robots.txt isn't suitable.

Change for the sake of it?

>>dbette+u2
Came here to say that, seems like nobody as the answer :/

Maybe they want to have finer details on page content, e.g: "you can index those pages but not those nodes" or "those nodes are also AI generated please ignore".

Otherwise I don't know, robots.txt is not sexy but definitely does the job.