AI discovery

Articles on AI discovery

1. llms.txt: what the spec says, what Common Crawl found, and what Google's AI guidance covers^{, 3 sources}

   The llms.txt proposal is a small markdown file with a defined format. Adoption is real, but none of the sources shows it affects AI search visibility.

   Footnotes DeskAEOOct 29, 20265 min read

2. robots.txt and llms.txt across 118 AEO tools and agency sites: what we found^{, 4 sources}

   The Footnotes Lab fetched robots.txt and llms.txt from every published tool and agency domain in our directory. Nearly all serve robots.txt, and very few block AI crawlers.

   Footnotes LabAI discoveryOct 29, 20264 min read

3. Cloudflare AI Crawl Control and pay per crawl: what they show and what the ratios miss^{, 3 sources}

   What a site owner can see and do with Cloudflare's AI Crawl Control, where pay per crawl stands, and how to read crawl-to-refer ratios.

   Footnotes DeskAI discoveryOct 29, 20264 min read

4. AI crawlers by purpose: what each fetches and the robots.txt lines that control it^{, 7 sources}

   OpenAI, Perplexity, Anthropic, Apple, Google and Common Crawl crawlers sorted by training, search and user-requested fetches, with the robots.txt rules for each.

   Footnotes DeskAI discoveryOct 29, 20264 min read