Robots.txt is where you tell crawlers (AI or otherwise) what should and shouldn't be read on your site.
Metadata like in tags, HTML meta tags, etc. is where you describe the content so meaning can be extracted from it by machines and automated processing.