llms.txt vs robots.txt vs sitemap.xml: which does what

These three files get discussed as though they were alternatives, and they are not. They answer different questions, they are read by different things, and two of them have been load-bearing for twenty-five years while the third is a proposal from 2024. Here is the whole distinction in one table, and then the detail.

robots.txt grants and refuses permission

The oldest of the three and the only one with real consequences if you get it wrong. It is a list of user agents and the paths each may or may not fetch. It is a request rather than a fence — it has never been anything else, and the same was true for search crawlers long before AI — but the major operators document the tokens they honour and do honour them.

240 of 293 (82%) of the sites we sampled publish one, which makes it the most widely adopted of the three by a wide margin.

The mistake that actually costs money is not omitting the file. It is writing one rule for "the AI bots" without noticing that the category contains crawlers with opposite effects on you: training crawlers, which cost you nothing to block, and answer engines, which are how a person ends up on your site from a chat window. Which is which is on the crawler reference.

sitemap.xml lists what exists

A machine-readable list of every URL you want found, with an optional last-modified date for each. It is a discovery aid, not an instruction: nothing is obliged to crawl what is in it, and nothing is prevented from crawling what is not. It matters most on large sites, on new sites with few inbound links, and on pages that are not reachable by following links from the homepage.

198 of 293 (68%) of the sample publish one.

The detail people skip is lastmod. A sitemap where every entry claims to have changed this morning is a sitemap that gets ignored, because the field stops carrying information the moment it is generated rather than recorded. If you cannot say when a page really changed, leave the field out.

llms.txt introduces the site

A Markdown document: your name, a paragraph an assistant can quote, then curated links with a sentence explaining each. Not every URL — the twenty you would send somebody who asked what you do. That curation is the entire difference from a sitemap, and the annotation is the part that does the work, because it lets an assistant choose the right page without fetching all of them.

No major assistant has publicly committed to reading it. That is the honest position and it does not change because the idea is appealing. What is true is that it costs an afternoon, has no runtime, and cannot hurt you — a cheap option on a convention that may become expected. Adopt it on that basis, not because somebody told you it gets you into ChatGPT.

65 of 293 (22%) of the sample publish one. The full argument, including what a good one looks like, is in the llms.txt piece.

Do you need all three?

Two of them, definitely. robots.txt because otherwise you have made no decision about crawler access and cannot tell anybody what your decision was. sitemap.xml because it is generated for free by every framework and static site host, and there is no argument against having one.

llms.txt is genuinely optional today, and anyone who tells you otherwise is ahead of the evidence. The order to do them in is the order of consequence: fix your robots.txt first, because that is the one that can currently be costing you traffic; publish a sitemap if you somehow do not have one; write the llms.txt when you have an afternoon.

What none of them do

None of these three files puts content in front of an assistant. If your page returns an empty container that fills in with JavaScript, or a challenge screen instead of your HTML, all three files are perfectly correct and the assistant still has nothing to read. The files describe the site; they do not rescue a page that cannot be read. That is a separate problem and usually the more expensive one.