Came here to say that, seems like nobody as the answer :/
Maybe they want to have finer details on page content, e.g: "you can index those pages but not those nodes" or "those nodes are also AI generated please ignore".
Otherwise I don't know, robots.txt is not sexy but definitely does the job.