A standard protocol for reputable crawlers to semantically understand some high-level page navigation rules.
Actual, useful crawling (i.e. to build search indices) would be much messier and more useless without most interesting sites putting up meaningful robots.txt guide-rails. Look at facebook.com/robots.txt and consider how much crap both Facebook and indexers would have to deal with lacking that information.