The state of Agent Experience, measured on 293 sites

There is a great deal of advice about making websites work for AI assistants and almost no measurement. So on 2026-09-03 we scanned 293 well-known public websites — SaaS, retail, news, documentation, government and reference — exactly the way a crawling assistant reads them: one plain HTTP request each, no browser, no JavaScript, a hard timeout. This is what came back.

No site is named here and none ever will be. Publishing a league table of businesses that never asked to be measured is the thing our consent rules exist to prevent, and the useful claim needs the count rather than the names. Every number below is regenerated from the repository by scripts/ax-study.mjs, so it can be checked rather than trusted.

The headline

83 of 293 (28%) score below 58, which is the point at which an assistant starts getting a page materially wrong rather than merely reading it thinly. At the other end, 12 of 293 (4%) score 85 or above.

The distribution matters more than the average. These are not obscure sites — they are companies with design teams, SEO budgets and performance dashboards. Whatever is going wrong is not going wrong through neglect; it is going wrong because nobody is measuring this reader.

Where the points actually go

The ten checks that fail most often across the sample, out of the 57 in the specification:

The pattern in that list is worth sitting with. These are not aesthetic failures or performance failures. They are failures to *state* things: a page that has a price but does not declare one, a company that has an address but does not publish it in a form anything can read.

The questions a customer asks, and how often they can be answered

Every scan asks whether a page can answer the questions a person actually puts to an assistant. Stated means machine-readable and quotable verbatim. Implied means it is there in prose or one fetch away — findable, paraphrasable, often not worth the request.

The gap between the first two columns is the single most useful finding in this study. On most of these sites the fact is there. A person reading the page can see the price, find the address, work out what is sold. It is simply not written in a form that survives being read by a machine — and the owner has no way of noticing, because to them the page looks complete.

Rendering: the failure with no error message

These sites ship an empty root element and assemble the page in the visitor's browser. Google Search has rendered JavaScript for years, so this costs them nothing in search and is invisible in every SEO tool they own. The crawlers behind AI assistants generally do not render, and so receive nothing.

Separately, 45 of 293 (15%) served something that looked like a bot wall rather than a page — a challenge from an edge product, delivered with a 200 status, logged nowhere as a failure.

What sites publish for agents

The two at the bottom are the interesting ones, and not because they are important yet. They are the cheapest possible differentiators: a file each, no runtime cost, and at these rates almost nobody in your industry has one.

Which crawlers are being turned away

We resolved each site's robots.txt against every named AI crawler individually. Blocking a training crawler is a legitimate decision. Blocking an answer engine removes you from results that would have linked to you — and the numbers suggest a great many sites did not distinguish between the two.

What a page costs a model to read

A context window is a fixed budget. A site that spends a third of every page on a mega-menu is paying that on every page an assistant ever reads of it, and the content it wanted read competes for what is left.

By category

What we would conclude from this

1. The problem is under-declaration, not bad content. The dominant failure is a fact that exists on the page in a form only a person can use. That is cheap to fix and almost nobody is fixing it.
2. The failure is silent by construction. No error, no alert, no dip in a dashboard. Every measurement tool most teams own is watching a different reader.
3. The agent-facing files are an open goal. At these publication rates, doing the obvious things puts a site ahead of nearly everyone in its category, for an afternoon of work.
4. Blocking is often accidental. The gap between training crawlers and answer engines is not widely understood, and one rule usually catches both.

The full rubric behind every number here is published in full and generated from the scoring code, so it cannot drift from what was actually measured. The distribution is on the AX Index.