robots.txt and llms.txt across 118 AEO tools and agency sites: what we found, 4 sources
The Footnotes Lab fetched robots.txt and llms.txt from every published tool and agency domain in our directory. Nearly all serve robots.txt, and very few block AI crawlers.
Footnotes LabAI discovery4 min read