> it can be super helpful for AI website crawlers, instead of using thousands of tokens to know what your website is about, they can do it with just a few hundred.
Why would the crawler trust you to be accurate instead of just figuring it out for itself?
Besides, they want to hoover up all the data for their training set anyway.