zlacker

> All a robots.txt is is a polite request to please follow the rules in it

At least in my country (Germany), respecting robots.txt is a legal requirement for data mining. See German Copyright Code, section 44b: https://www.gesetze-im-internet.de/urhg/__44b.html

(IANAL)