The state of Agent Experience, measured on 292 sites

There is a great deal of advice about making websites work for AI assistants and almost no measurement. So on 2026-09-06 we scanned 292 well-known public websites — SaaS, retail, news, documentation, government and reference — exactly the way a crawling assistant reads them: one plain HTTP request each, no browser, no JavaScript, a hard timeout. This is what came back.

No site is named here and none ever will be. Publishing a league table of businesses that never asked to be measured is the thing our consent rules exist to prevent, and the useful claim needs the count rather than the names. Every number below is regenerated from the repository by scripts/ax-study.mjs, so it can be checked rather than trusted.

The headline

82 of 292 (28%) score below 58, which is the point at which an assistant starts getting a page materially wrong rather than merely reading it thinly. At the other end, 8 of 292 (3%) score 85 or above.

The distribution matters more than the average. These are not obscure sites — they are companies with design teams, SEO budgets and performance dashboards. Whatever is going wrong is not going wrong through neglect; it is going wrong because nobody is measuring this reader.

Where the points actually go

The ten checks that fail most often across the sample, out of the 70 in the specification:

The pattern in that list is worth sitting with. These are not aesthetic failures or performance failures. They are failures to *state* things: a page that has a price but does not declare one, a company that has an address but does not publish it in a form anything can read.

The questions a customer asks, and how often they can be answered

Every scan asks whether a page can answer the questions a person actually puts to an assistant. Stated means machine-readable and quotable verbatim. Implied means it is there in prose or one fetch away — findable, paraphrasable, often not worth the request.

The gap between the first two columns is the single most useful finding in this study. On most of these sites the fact is there. A person reading the page can see the price, find the address, work out what is sold. It is simply not written in a form that survives being read by a machine — and the owner has no way of noticing, because to them the page looks complete.

Rendering: the failure with no error message

These sites ship an empty root element and assemble the page in the visitor's browser. Google Search has rendered JavaScript for years, so this costs them nothing in search and is invisible in every SEO tool they own. The crawlers behind AI assistants generally do not render, and so receive nothing.

Separately, 45 of 292 (15%) served something that looked like a bot wall rather than a page — a challenge from an edge product, delivered with a 200 status, logged nowhere as a failure.

What sites publish for agents

The two at the bottom are the interesting ones, and not because they are important yet. They are the cheapest possible differentiators: a file each, no runtime cost, and at these rates almost nobody in your industry has one.

Which crawlers are being turned away

We resolved each site's robots.txt against every named AI crawler individually. Blocking a training crawler is a legitimate decision. Blocking an answer engine removes you from results that would have linked to you — and the numbers suggest a great many sites did not distinguish between the two.

What a page costs a model to read

A context window is a fixed budget. A site that spends a third of every page on a mega-menu is paying that on every page an assistant ever reads of it, and the content it wanted read competes for what is left.

What survives being cut into passages

An assistant is not handed your page. It is handed a passage of a few hundred tokens that a retrieval system cut out of it, next to passages from other sites, and the answer is written from that. So we ran the cut on every page in the sample and asked of each passage whether somebody who saw only it would know what it was about.

The median page survives it, and that is worth saying even though it is the less interesting answer: these are front pages of well-known companies, which are short, heavily headed and written to be skimmed. The tail is where the finding is. 17 of 261 (7%) leave under three fifths of their content in a passage that could be quoted, 23 of 261 (9%) are more than half navigation once cut up, and 5 of 261 (2%) have no prose passages at all.

Pages that disagree with themselves

A page speaks twice: once in prose an assistant has to interpret, and once in structured data it can simply believe. Nobody proof-reads the second, because nobody can see it. 37 of 292 (13%) of the sites in this sample publish at least one fact in their structured data that the page itself does not support, and 3 of 292 (1%) state something that outright contradicts what a reader can see.

Most of these are contact details: a telephone number or an email address published in the structured data and printed nowhere a visitor could check it. That is the shape a stale copy-paste leaves behind, and it is the number an assistant reads out when somebody asks how to get in touch.

Structured data that resolves to nothing

A type name that is not in the schema.org vocabulary is not a slightly wrong type. It is no type at all: the consumer keys the node on it, fails to resolve it, and discards everything underneath. The page carries the maintenance cost of the block and gets none of the benefit, and nothing anywhere reports it — the JSON is valid, the context is right, and a browser shows the page exactly as intended.

That is a smaller number than the rest of this study, and the reason is worth saying: these are famous companies, and the thing this catches is a typo. 0 of 292 (0%) declare a type that is not in the vocabulary, which would cost them the whole block it is on. 10 of 292 (3%) use a property that is not, which costs them that one fact. It is the failure with the worst ratio of consequence to effort on the whole list — nothing reports it, a browser shows the page exactly as intended, and the fix is one character.

The pattern in what turns up is instructive. Most of them are a plural where the vocabulary is singular, or a capital letter where it is lower case — a name that reads correctly to a person and resolves to nothing at all:

The cheapest thing on this list, and half the web has not done it

Crawlers revisit. With one of those headers a crawler asks "has this changed?" and receives an empty 304; with neither it downloads the whole page again every time it wonders. It costs one line of server configuration, most static hosts do it already, and it is the single most effective thing a site complaining about AI crawler traffic could do about it.

By category

What we would conclude from this

1. The problem is under-declaration, not bad content. The dominant failure is a fact that exists on the page in a form only a person can use. That is cheap to fix and almost nobody is fixing it.
2. The failure is silent by construction. No error, no alert, no dip in a dashboard. Every measurement tool most teams own is watching a different reader.
3. The agent-facing files are an open goal. At these publication rates, doing the obvious things puts a site ahead of nearly everyone in its category, for an afternoon of work.
4. Blocking is often accidental. The gap between training crawlers and answer engines is not widely understood, and one rule usually catches both.
5. Almost nobody has thought about the passage. Getting the page fetched and understood is where every existing tool stops, and it is the first half. What happens after the cut decides whether any of it can be quoted, and the tail of that distribution is worse than the median suggests.

The full rubric behind every number here is published in full and generated from the scoring code, so it cannot drift from what was actually measured. The distribution is on the AX Index.