zlacker

The point of robots.txt is to inform well behaved scrapers about how to behave. It is not designed nor intended to prevent bad actors.

Which is good design: don't pretend to solve problems you can't.