llms.txt: what the spec says, what Common Crawl found, and what Google's AI guidance covers

What the proposal specifies

The proposal says: "We propose adding a /llms.txt markdown file to websites to provide LLM-friendly content."¹ The file can sit at the site root or at any path, and it covers the pages under that path.¹ A file at /docs/llms.txt, for example, covers /docs/.¹ Where more than one file applies, agents should use the most specific one.¹

The file uses markdown to structure the information rather than a classic structured format such as XML.¹ The stated reason is that many of these files are expected to be read by language models and agents.¹ Its content still follows a specific format that standard programmatic tools can read.¹

The spec sets a fixed order of sections.¹ A byte-order mark is optional.¹ An H1 with the name of the project or site is the only required section.¹ Next in the order comes a blockquote with a short summary of the project.¹

After that come markdown sections of any type except headings, with more detailed information about the project.¹ File lists live in sections delimited by H2 headers.¹ Each entry in a file list needs a markdown hyperlink, optionally followed by a colon and notes.¹ By convention, the "Optional" section holds secondary information: links an agent can skip when a shorter context is needed.¹

The proposal has a second half that is easy to miss. It proposes that pages agents might need provide a clean markdown version at the same URL as the original page, with .md appended or the extension replaced by .md.¹ It also recommends standard link relations: rel="alternate" type="text/markdown" for the markdown version, and rel="describedby" for the llms.txt file that covers a page.¹ The FastHTML project follows both proposals for its documentation.¹ Whether you also publish markdown page versions is a separate decision from publishing the file.

How it relates to robots.txt and sitemaps

The proposal says llms.txt is designed to coexist with current web standards.¹ Sitemaps list all pages for search engines, while llms.txt "offers a curated overview for LLMs."¹ A sitemap often will not have the LLM-readable versions of pages listed.¹

A sitemap also does not include URLs to external sites, even though they might help an agent understand the information.¹ It will generally cover documents too large to fit in an LLM context window.¹

On robots.txt, the proposal says the file lets automated tools know what access to a site is considered acceptable, such as for search indexing bots.¹ The llms.txt file is instead used on demand, when an agent needs information about a topic while assisting a user.¹ Agents are expected to view or search the file to find the information they need, then follow the relevant links.¹ Treat it as a guide to your content, not as a way to control access.

The proposal also states what it expected. It thought llms.txt would mainly be useful for inference rather than training, and says that is how it has been used.¹ It adds that use is heaviest in software documentation, where coding agents follow the file to find API references and tutorials.¹

Who ships it

The proposal's author says that "thousands of sites publish an llms.txt file."¹ The AI labs themselves publish llms.txt files for their own developer docs.¹ These are the proposal's own adoption claims, so read them as the author's account.

The proposal lists platforms and plugins that generate the file automatically.¹ Yoast SEO is a WordPress plugin that generates and maintains one.¹ Wix generates one for every Wix site.¹ Mintlify generates llms.txt and markdown page versions for every site it hosts.¹

What Common Crawl found

Common Crawl published a content analysis of llms.txt files from its July 2026 crawl archive.² It examined 584,107 files.² Its summary has three findings:

• Two thirds of the files are produced by a plugin.²
• Half follow the structure the specification defines.²
• A few files contain prompt injections.²

Only the blog's summary is available here, so finer breakdowns are not reported. Treat a plugin-generated file as something to review, not as finished work. Check your own file, especially an auto-generated one, for text addressed to the model rather than to a reader.

What Google says

Google's guide on AI features covers AI Overviews and AI Mode.³ To appear as a supporting link, "a page must be indexed and eligible to be shown in Google Search with a snippet."³ The guide adds: "There are no additional technical requirements."³ It also says that "specific optimization isn't required for AI Overviews and AI Mode."³

The guide lists existing fundamentals, including allowing crawling in robots.txt and by any CDN or hosting infrastructure.³ It also lists making sure important content is available in textual form.³

On control, the guide says AI is built into Search and integral to how Search functions.³ Robots.txt directives for Googlebot are the control for site owners to manage how their sites are crawled for Search.³ The guide offers controls to limit the information shown from your pages in Search.³ For AI training and grounding in some of Google's other systems, it points to Google-Extended.³

Read the guide for what it says, not for what it leaves out. Do not infer a Google position on llms.txt from it, in either direction.

Should you ship one?

The evidence supports a narrow answer. The proposal defines a format and describes how agents are expected to use it.¹ Common Crawl analysed 584,107 llms.txt files from its July 2026 crawl, and half follow the structure the specification defines.² Google's guide on its own AI features says there are no additional technical requirements.³ None of the sources shows that an llms.txt file raises citations or rankings in any engine, so do not promise that to a client or a boss.

A practical way to decide:

• Documentation or API sites: ship one. The proposal says llms.txt files are used most heavily for software documentation.¹ Check whether your docs platform already generates one.
• Marketing and content sites: treat it as optional. Google ties eligibility as a supporting link to a page being indexed and eligible to be shown with a snippet.³ Put that work first.
• Auto-generated files: open the output and read it. Remove anything you would not want an agent to treat as authoritative, and check the structure against the spec.
• Hand-written files: keep them short and curated, with brief descriptions on each link.

If you ship one, test it the way the proposal suggests: ask an agent questions about your content, giving it only your llms.txt as a starting point.¹ If the answers are wrong, the file needs fixing. If they are right, you have a working guide for agents. That is a modest outcome, and it is the one the sources support.