zlacker

Yeah, and if you make it that complex to extract consent you won't get any. Think one step ahead maybe. One switch, one thing to parse.

>>Rambli+(OP)
Robots.txt is where you tell crawlers (AI or otherwise) what should and shouldn't be read on your site.

Metadata like in tags, HTML meta tags, etc. is where you describe the content so meaning can be extracted from it by machines and automated processing.

>>Rambli+(OP)
1. OP said “what it is about, when was it published, the author, etc.” That’s what these mechanisms already cover. Consent is an interesting possibility that I’ll admit something like ai.txt might be better for, but my post was largely focused on the OP.

2. These are all complex formats. If you want to ingest and process them then you already have to build all the hard parts. Getting the metadata out is dead simple compared to parsing, decoding, and then processing an image, for example.