"Your do not enter sign uses font we don't like, so we'll just ignore it"
The intent of robots.txt is to help crawlers, for example, to keep crawlers from getting stuck in a recursive loop of dynamic pages, or from crawling pages with no value. robots.txt is not for banning, restricting, or hindering crawlers.
I'm honestly surprised they're required to abstain from doing so at the author's request.
You can only read the context of the match after finding the search result after all, not the whole book.
It's an example of significant overreach of intellectual property from how I see it. The robot.txt rational doesn't apply there either, as their scanning does not impact anyone's resources. And it's been published, which makes it public by definition.
That's not to say that I disagree. In most cases robots.txt is not legally binding. It only becomes a legal danger to not follow it when the person running the site has power and can buy a DA to indict you.
Which is of course not the real reason.
The reason Google doesn't follow the robots.txt protocol is (1) they don't want to (2) they can get away with it.
They do (or did). They showed the text around the search term, around a page or so, which made it possible to reconstruct the whole book without that much effort.
Speaking of this and other cases of trying to punish someone for every iteration of a for loop - I wonder if the result would be the same if the accused drove actual browser to click stuff in a for loop, vs. using curl directly. I imagine the same, but then...
... what if they paid N people some token amount of money, to have each of those people do one step of the loop and send them the result? Does executing a for loop entirely on in part on the human substrate instead of in silico is seen as abuse under CFAA?
(I have a feeling that it might not be - there's lots of jobs online and offline that involve one company paying lots of people some money for gathering information from their competitors, in a way the latter very much don't like.)