llms.txt Explained: What It Is and Which AI Crawlers Actually Read It

llms.txt is a markdown file at a site's root that points AI models to its most important pages, but no major crawler — not OpenAI's GPTBot, not Google's Gemini, not Anthropic's Claude — has publicly confirmed that it actually reads or uses one, at least based on currently available evidence. The file was proposed in 2024 as a fix for limited context windows, letting sites hand models a curated summary instead of raw HTML. Adoption by tool vendors and documentation sites appears to be real and growing. Adoption by the crawlers that decide what gets cited is not, at least not yet, as far as can be confirmed.

What llms.txt is and where the idea came from

llms.txt is a plain-text markdown file, placed at a website's root, meant to give large language models a curated summary of a site's most important content. Jeremy Howard, co-founder of Answer.AI, proposed the format in September 2024 and published the spec at llmstxt.org.

The problem Howard was trying to solve is a real one. LLMs work with a limited context window, and most websites bury their useful content under navigation menus, JavaScript widgets, ads, and boilerplate that models have to wade through just to find a paragraph of substance. Howard's proposal borrows a trick web developers already know from documentation tools: strip the noise, keep the markdown, and hand the model a clean list of links with one-line descriptions. The file structure he suggested is deliberately minimal — an H1 title, a short blockquote summary, then H2 sections grouping links by topic (docs, guides, API references, and so on).

It's worth separating the idea from the file itself. Howard also proposed a companion convention, llms-full.txt, that inlines the entire content rather than just linking to it, aimed at cases where a model can pull in a much larger context. Both live under the same llms.txt umbrella but solve slightly different problems: one is a curated index, the other is a full-text dump.

The comparison people reach for instinctively is robots.txt, the 1994 Robots Exclusion Protocol that tells crawlers what they're *not* allowed to fetch. Llms.txt does the opposite job — it's an invitation, not a restriction, and it's not enforced by any protocol or governing body. No W3C standard, no IETF RFC, nothing that forces a crawler to look for it. Any site can drop a file at /llms.txt today; whether any model actually reads it is a separate question, and one this article answers section by section. If you're trying to understand how this fits into the broader picture of getting cited by AI systems, it's worth reading up on what AEO actually is before going deeper here.

That distinction matters for expectations. Robots.txt and sitemap.xml are backed by decades of crawler convention and are checked by essentially every search bot. Llms.txt is roughly a year old, unratified, and voluntary on the crawler side — so treat it as an experimental content aid, not a mechanism you can rely on for guaranteed visibility.

Takeaway: llms.txt is a voluntary, unstandardized markdown file — proposed by Jeremy Howard in 2024 — that summarizes a site's key content for LLMs, but no protocol requires any crawler to fetch it.

The exact syntax and file structure of llms.txt

llms.txt is a plain Markdown file placed at your domain's root (/llms.txt) that lists links and short descriptions meant to help language models understand a site without parsing full HTML pages. The spec comes from Jeremy Howard and the Answer.AI team, who published it in September 2024 as a proposed convention, not an official web standard.

The structure follows a fixed order:

• **H1 title** — the project or site name, required, one line
• **Blockquote summary** — a one- or two-sentence description directly under the title, using standard Markdown `>` syntax
• **Optional context paragraphs** — plain prose giving background, with no fixed heading
• **H2 sections with linked lists** — each H2 groups related pages as Markdown links (`[Title](URL): description`)
• **An "Optional" H2** — by convention, this holds secondary links a model can skip if it's short on context

Here's a minimal example:

# Acme Docs

> Acme is a tool for X. This file lists our core documentation for LLMs.

## Docs

- [Getting Started](https://acme.com/start): setup and installation
- [API Reference](https://acme.com/api): full endpoint list

## Optional

- [Changelog](https://acme.com/changelog): release history

Two things trip people up. First, llms.txt isn't a directive file like robots.txt — there's no syntax for allowing or blocking crawlers. It's purely descriptive: a curated table of contents, not an access-control rule. Second, the spec also defines llms-full.txt, a companion file containing the full text content of key pages inline, rather than just links to them. Some sites publish both — llms.txt as an index, llms-full.txt as a single concatenated document for models that fetch it directly.

Validation is loose since there's no registry or W3C body enforcing it — you're following Markdown conventions and the informal spec at llmstxt.org, not a checked schema. Tools such as the llms.txt generator on GitHub can reportedly help scaffold the file, but nothing verifies compliance server-side.

Takeaway: llms.txt is a Markdown file with a fixed title/summary/H2-link structure — correct syntax matters for readability, but no enforcement mechanism exists to check it.

llms.txt vs robots.txt vs sitemap.xml: how they differ

llms.txt curates content for AI models to read, robots.txt blocks or allows crawler access, and sitemap.xml lists every URL for search engine indexing. Each file talks to a different audience with a different verb: llms.txt *recommends*, robots.txt *permits*, sitemap.xml *enumerates*.

The mechanics separate them further:

• **robots.txt** follows the [Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html), now an official IETF standard. It uses `User-agent` and `Disallow` rules to control crawler access, and every major bot checks it before fetching anything.
• **sitemap.xml** is an XML inventory of your URLs, defined at [sitemaps.org](https://www.sitemaps.org/), that search engines use for discovery and recrawl scheduling. It enumerates everything, with no curation.
• **llms.txt** is a curated markdown reading list for language models. No standards body has ratified it, and no major crawler has confirmed fetching it.

In practice the three complement each other: robots.txt for access control, sitemap.xml for coverage, llms.txt as a low-cost bet on where AI crawlers might go next.

Takeaway: robots.txt controls access, sitemap.xml enumerates URLs, llms.txt curates content — only the first two are backed by standards crawlers actually follow.

Should you ship one?

Yes — the cost is about fifteen minutes and the downside is zero. The honest state of play: documentation platforms like Mintlify generate llms.txt automatically for hosted docs, and a growing list of tool vendors publish the file, while OpenAI, Google, Anthropic, and Perplexity have made no commitment to reading it. Some site owners report AI user-agents occasionally fetching /llms.txt in server logs, but sporadic fetches are not a citation pipeline.

Treat it like early-days schema markup: adopt cheaply, keep it accurate, and skip the miracle expectations. If AI engines do start honoring the convention, sites with a clean llms.txt are first in line; if they never do, you've lost fifteen minutes. Seenbot publishes its own at seenbot.cc/llms.txt — and whether your pages actually surface in AI answers, with or without the file, is exactly what Seenbot measures. For the broader playbook, see how to get your site cited by ChatGPT and Perplexity.