I don't know that this is true for the US. As far back as I can remember, there have been questions about whether a robots.txt file means you don't have permission to engage in those activities. The CFAA is one law that has repeatedly come up. See for example https://www.natlawreview.com/article/doj-revises-policy-cfaa...
It might be the case that there is nothing there legally, but I don't think I'd describe the actions of search engines as being driven by a moral imperative.