Writing
What can be said about this from measurement rather than opinion?
Everything here is written because somebody types that question into a search box and gets speculation back. Where a claim can be measured it is measured, on a sample of the real web, and the numbers behind it are regenerated from the repository rather than typed in by hand.
Agent files
- llms.txt vs robots.txt vs sitemap.xml: which does what Three text files at your domain root, constantly confused. One grants permission, one lists every URL, one introduces the site — and only two are read by anything today.
- llms.txt: what it is, and whether anyone is actually using it A plain-text map of your site written for AI assistants rather than browsers. Here is the format, what it does and does not buy you, and how many real sites publish one.
Crawler access
- Your bot protection is blocking the assistants that would cite you Of 293 well-known websites we fetched, 45 refused a plain HTTP request outright. Almost none of them meant to.
- The AI crawlers in your robots.txt, and the one that costs you customers Some AI crawlers train on your content. Others fetch you at the moment a customer asks a question, and cite you in the answer. Most sites block both with one rule and never find out.
What is Agent Experience?
Agent Experience, or AX, is how well a website works for the software that reads it on a person's behalf — an AI assistant answering a question, a crawler building an index, a shopping agent comparing prices. It is a separate question from how the site looks or how fast it loads, because those readers never see the design and usually never run the JavaScript.
Does llms.txt actually do anything yet?
No major assistant has publicly committed to reading llms.txt, and anyone telling you otherwise is guessing. It costs about ten minutes to publish and it is the cheapest way to state, in one place, what your site is and where the pages that matter are — but it should be adopted as a low-cost bet, not as a fix for anything measurable today.
Which AI crawlers should I allow in robots.txt?
The distinction worth making is between crawlers that train models and crawlers that fetch a page live to answer somebody who is asking about you right now. Blocking the second kind removes you from the answer. Most sites that block AI traffic have blocked both without noticing the difference.
Why can an AI assistant not read my React site?
Because most assistants make one plain HTTP request and read what comes back, without running JavaScript. A single-page application returns an empty div and a script tag, so the assistant sees a page with no content on it. The fix is server-side rendering or pre-rendering for the pages that carry your actual information.
feed — the form that hands an agent the whole stream without crawling for it, which is the same thing this scanner looks for on your site.