{"article":{"title":"robots.txt and llms.txt across 118 AEO tools and agency sites: what we found","path":"/lab/robotstxt-and-llmstxt-across-118-aeo-tools-and-agency-sites-what-we-found"},"name":"directory-robots","sites":[{"domain":"adobe.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"ahrefs.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"airops.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"algorythmic.co","robots":true,"blocked":[],"llmsTxt":true},{"domain":"amicited.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"amsive.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"animalz.co","robots":true,"blocked":[],"llmsTxt":false},{"domain":"athenahq.ai","robots":true,"blocked":[],"llmsTxt":true},{"domain":"averi.ai","robots":true,"blocked":[],"llmsTxt":true},{"domain":"beomniscient.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"bing.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"bluefishai.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"botify.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"brainlabsdigital.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"brandlight.ai","robots":true,"blocked":[],"llmsTxt":true},{"domain":"brightedge.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"clearscope.io","robots":true,"blocked":[],"llmsTxt":false},{"domain":"cloudflare.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"cognizo.ai","robots":true,"blocked":[],"llmsTxt":true},{"domain":"conductor.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"conversion.com.br","robots":true,"blocked":[],"llmsTxt":true},{"domain":"dept.global","robots":true,"blocked":[],"llmsTxt":false},{"domain":"directiveconsulting.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"duwerk.de","robots":true,"blocked":[],"llmsTxt":false},{"domain":"evertune.ai","robots":true,"blocked":[],"llmsTxt":true},{"domain":"exposureninja.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"firstpagesage.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"frase.io","robots":true,"blocked":[],"llmsTxt":true},{"domain":"geoptie.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"geosurge.ai","robots":true,"blocked":[],"llmsTxt":true},{"domain":"gitbook.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"github.com","robots":true,"blocked":["Bytespider"],"llmsTxt":true},{"domain":"gofishdigital.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"google.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"graphite.io","robots":true,"blocked":[],"llmsTxt":false},{"domain":"gsqi.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"higoodie.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"huble.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"hubspot.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"impressiondigital.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"infidigit.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"inlinks.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"interodigital.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"ipullrank.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"kevin-indig.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"knowatoa.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"knownagents.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"llmpulse.ai","robots":true,"blocked":[],"llmsTxt":true},{"domain":"llmrefs.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"lovedby.ai","robots":true,"blocked":[],"llmsTxt":true},{"domain":"mangools.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"mariehaynes.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"mintlify.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"minuttia.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"nightwatch.io","robots":true,"blocked":[],"llmsTxt":false},{"domain":"nogood.io","robots":true,"blocked":[],"llmsTxt":true},{"domain":"omnius.so","robots":true,"blocked":[],"llmsTxt":true},{"domain":"oncrawl.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"orainti.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"otterly.ai","robots":true,"blocked":[],"llmsTxt":true},{"domain":"peakace.agency","robots":true,"blocked":[],"llmsTxt":false},{"domain":"peec.ai","robots":true,"blocked":[],"llmsTxt":false},{"domain":"prerender.io","robots":true,"blocked":[],"llmsTxt":true},{"domain":"primelis.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"promptmonitor.io","robots":true,"blocked":[],"llmsTxt":true},{"domain":"promptwatch.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"prosperitymedia.com.au","robots":true,"blocked":[],"llmsTxt":false},{"domain":"radyant.io","robots":true,"blocked":[],"llmsTxt":true},{"domain":"rankmath.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"rankprompt.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"rankscale.ai","robots":true,"blocked":[],"llmsTxt":true},{"domain":"rankshift.ai","robots":true,"blocked":[],"llmsTxt":false},{"domain":"rebootonline.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"relixir.ai","robots":false,"blocked":[],"llmsTxt":false},{"domain":"revenuezen.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"rocktherankings.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"scalenut.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"schema.org","robots":true,"blocked":[],"llmsTxt":false},{"domain":"schemaapp.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"screamingfrog.co.uk","robots":true,"blocked":[],"llmsTxt":false},{"domain":"scrunch.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"search-console.ai","robots":true,"blocked":[],"llmsTxt":false},{"domain":"searchable.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"searchatlas.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"seerinteractive.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"semetrical.com","robots":true,"blocked":["GPTBot","ClaudeBot","PerplexityBot","Google-Extended","Applebot-Extended","CCBot"],"llmsTxt":false},{"domain":"semrush.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"seoclarity.net","robots":true,"blocked":[],"llmsTxt":false},{"domain":"seoprofy.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"seoworks.co.uk","robots":true,"blocked":[],"llmsTxt":true},{"domain":"seranking.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"siegemedia.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"similarweb.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"singlegrain.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"siteline.ai","robots":true,"blocked":[],"llmsTxt":false},{"domain":"spyglasses.io","robots":true,"blocked":[],"llmsTxt":false},{"domain":"stellarising.com","robots":false,"blocked":[],"llmsTxt":false},{"domain":"studiohawk.com.au","robots":true,"blocked":[],"llmsTxt":true},{"domain":"superlines.io","robots":true,"blocked":[],"llmsTxt":true},{"domain":"surfer.ai","robots":true,"blocked":[],"llmsTxt":false},{"domain":"thepublive.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"thriveagency.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"tjrobertson.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"tollbit.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"trakkr.ai","robots":true,"blocked":[],"llmsTxt":true},{"domain":"tripledart.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"tryprofound.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"victorious.com","robots":true,"blocked":[],"llmsTxt":false},{"domain":"waikay.io","robots":true,"blocked":[],"llmsTxt":false},{"domain":"webfx.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"withgauge.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"wordlift.io","robots":true,"blocked":[],"llmsTxt":true},{"domain":"wordpress.org","robots":true,"blocked":[],"llmsTxt":true},{"domain":"writesonic.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"xpanddigital.io","robots":true,"blocked":[],"llmsTxt":false},{"domain":"yext.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"yoast.com","robots":true,"blocked":[],"llmsTxt":true},{"domain":"ziptie.ai","robots":true,"blocked":[],"llmsTxt":true}],"agents":[{"share":"0.9%","token":"GPTBot","purpose":"training","blocking":1,"operator":"OpenAI"},{"share":"0.0%","token":"OAI-SearchBot","purpose":"search","blocking":0,"operator":"OpenAI"},{"share":"0.0%","token":"ChatGPT-User","purpose":"user request","blocking":0,"operator":"OpenAI"},{"share":"0.9%","token":"ClaudeBot","purpose":"training","blocking":1,"operator":"Anthropic"},{"share":"0.0%","token":"Claude-SearchBot","purpose":"search","blocking":0,"operator":"Anthropic"},{"share":"0.0%","token":"Claude-User","purpose":"user request","blocking":0,"operator":"Anthropic"},{"share":"0.9%","token":"PerplexityBot","purpose":"search","blocking":1,"operator":"Perplexity"},{"share":"0.0%","token":"Perplexity-User","purpose":"user request","blocking":0,"operator":"Perplexity"},{"share":"0.9%","token":"Google-Extended","purpose":"training","blocking":1,"operator":"Google"},{"share":"0.9%","token":"Applebot-Extended","purpose":"training","blocking":1,"operator":"Apple"},{"share":"0.9%","token":"CCBot","purpose":"training","blocking":1,"operator":"Common Crawl"},{"share":"0.9%","token":"Bytespider","purpose":"training","blocking":1,"operator":"ByteDance"},{"share":"0.0%","token":"meta-externalagent","purpose":"training","blocking":0,"operator":"Meta"},{"share":"0.0%","token":"Amazonbot","purpose":"training","blocking":0,"operator":"Amazon"}],"method":"On 2026-10-29 we requested https://<domain>/robots.txt and https://<domain>/llms.txt for the 118 distinct domains of the published tools and agencies in the AEO Footnotes directory, following redirects, with a 12 second timeout and the user agent AEOFootnotesBot. A robots.txt counts when it answers 200 with a text body. A crawler counts as blocked when the robots.txt group that applies to it under RFC 9309 (its own user-agent group, else the * group) disallows / without allowing /. An llms.txt counts when it answers 200 with a body that starts with a markdown H1, as the proposal specifies. Shares are of the sites that serve a robots.txt.","population":118,"withRobots":116,"collectedAt":"2026-10-29","withLlmsTxt":64}