llms.txt: what it is, and whether anyone is actually using it

An llms.txt file is a Markdown document at the root of your domain that tells an AI assistant what your site is and which pages matter. It sits alongside robots.txt and sitemap.xml and does a different job from both: robots.txt grants and refuses permission, sitemap.xml lists every URL you have, and llms.txt is the short, curated version — the twenty links you would send somebody who asked what you do.

The idea was proposed by Jeremy Howard of Answer.AI in September 2024. It is a convention, not a standard: there is no committee behind it and no specification you can be non-compliant with.

What it looks like

One h1 with your name, a blockquote summarising what you do, then sections of annotated links. That is the whole format.

Copy# Acme Robotics

> Acme builds warehouse robots for companies that would rather not buy a warehouse.
> Founded 2019, based in Rotterdam, shipping to the EU and the UK.

## Product
- [How it works](https://acme.test/product): The arm, the rail, and what fits on them.
- [Pricing](https://acme.test/pricing): Three plans, exact prices, no "contact us".

## Docs
- [Quickstart](https://acme.test/docs/start): Running in about ten minutes.
- [API reference](https://acme.test/docs/api): Every endpoint, with examples.

## Policies
- [Returns](https://acme.test/returns): 30 days, we pay the shipping.
- [Contact](https://acme.test/contact): support@acme.test, answered within a day.

The sentence after each link is the part people skip and the part that does the work. A link with no annotation is a sitemap entry wearing different syntax; the annotation is what lets an assistant choose the right page without fetching all of them.

Does anything actually read it?

This is the question the rest of the internet is strangely quiet about, so here is the honest answer: no major assistant has publicly committed to reading llms.txt. OpenAI, Anthropic and Google have not announced support for it. Anybody telling you that publishing one gets you into ChatGPT is selling something.

What is true is narrower and still worth having. The file is fetched by a growing number of agent frameworks and documentation tools; some products consume it directly; and it costs you one afternoon and no runtime. It is a cheap option on a convention that may become expected, taken out at a moment when almost nobody else has taken it.

That last part is measurable, so we measured it. Of the 293 well-known public sites in our sample — SaaS, retail, news, documentation, government and reference — this many publish an llms.txt:

For comparison, 198 of 293 (68%) publish a sitemap.xml and 240 of 293 (82%) publish a robots.txt. Whatever else is true of llms.txt, it is not yet a thing your competitors have done.

Why we score it, and what that means

AXRAY gives llms.txt fourteen points, which is the heaviest single check in the Agent Contract pillar. It is fair to ask why a file nobody has committed to reading is worth that much, and the answer is not "because we think it will catch on".

It is that writing one forces the exercise. To produce a useful llms.txt you have to decide, in one sentence, what your site is for, and then choose the twenty pages that matter out of the four hundred you have. Almost every site that does this discovers something wrong with its own information architecture on the way. The file is the artefact; the thinking is the value.

We also do not give it away. An empty or link-free file earns at most half the points and the report labels it thin evidence, because publishing a hollow artefact to satisfy a check is the failure mode a public rubric invites.

How to write one that is not useless

1. Serve it at https://yourdomain/llms.txt as text/plain or text/markdown. Not behind a redirect, not behind a login.
2. Start with # Your name, then a blockquote of two or three sentences a stranger could quote about you verbatim. Write it as though it will be read out loud, because it will be.
3. Group links under two to five ## headings that reflect intent — Product, Docs, Policies — rather than your navigation structure.
4. Give every link a sentence saying what is on that page. This is the whole point of the format.
5. Keep it under about fifty links. It is a table of contents, not an index.
6. Use absolute URLs. A relative link in a file that gets copied into a prompt resolves to nothing.
7. Update it when the site changes. A stale map is worse than no map, because it will be believed.

The three mistakes we see most

Pasting the sitemap. Four hundred bare URLs is the thing llms.txt exists as an alternative to. If it needs a scroll bar, it is the wrong document.

Leaving the template in. Files still pointing at example.com are common, because someone copied a snippet from a blog post and shipped it. Our own scanner used to hand out such a snippet, and our own check marked people down for using it; we now generate a draft from the site being scanned instead.

Writing marketing copy. The blockquote is not a tagline. "We empower teams to unlock synergy" tells an assistant nothing it can repeat to a customer. "We sell warehouse robots to European mid-size logistics companies, from €40,000" does.

What to do next

Check whether you have one, and whether what you have says anything. A scan reads your llms.txt, counts its sections and its usable links, and — if it is missing or hollow — drafts one from your own site, using your own name, description and navigation rather than a template.