Robots.txt have served the simple purpose of directing bots like Google to the different parts of your website since the beginning of internet time.
They still serve the same purpose, they tell bots where to go, and most importantly, they tell bots how to find your site map.
Robots.txt is not there to prevent malicious crawlers from accessing pages as you have suggested.
The robots.txt file acts simply like a garden gate. The good and honest people will honor the gate, while the more malicious might ignore it and hop the fence or something.