AXRAY
Sign inCreate account
Menu

Open dataset

The state of Agent Experience, 2026

292 well-known public websites, each fetched once the way an AI agent fetches it and scored against a published rubric. The full data is below, downloadable, and free to reuse with attribution.

What it found

68/100 median score across the sample
8 sites out of 292 scoring above 85
45 refuse AI crawlers outright in robots.txt
72 serve a bot wall instead of their content
11 send almost nothing without JavaScript
64 publish an llms.txt

The short version: the median site scores 68 out of 100, only 8 of 292 clear 85, and the commonest failures are not exotic. They are a bot wall in front of the content, a page that needs JavaScript to say anything, and structured data that is absent rather than broken. The written analysis goes through it in full.

Download it

One JSON file: the score distribution, per-check pass and fail counts for all 70 checks, per-crawler robots.txt outcomes, artefact adoption, retrieval measurements and the structured-data contradictions.

Download the dataset (JSON)

Licensed CC BY 4.0. Use it, quote it, redraw the charts — the only condition is attribution.

How it was measured

  • Sample: 292 well-known public websites across the categories in the data file. 11 could not be fetched at all and are reported separately rather than scored as zero.
  • Fetch: one plain HTTP GET per site from a single origin. No browser, no JavaScript execution, no retries, no cookies — because that is what most agents do.
  • User agent: we identify ourselves honestly as AXRAY/1.0 and obey robots.txt. A study that measured crawler access by pretending to be a browser would be measuring the wrong thing.
  • Scoring: version 1.5.1 of the published specification, with every weight public. Re-run it yourself with npx axray-cli and you should get the same number.
  • Date: 2026-09-06. The web moves; treat it as a snapshot.

What it does not show

Worth stating, because a dataset gets quoted long after its caveats are forgotten.

  • No site is named. This is a distribution, not a league table. Nothing about an individual site appears publicly unless its owner verifies the domain and asks to be listed.
  • It is one fetch from one place. A site that geo-varies, rate-limits, or was having a bad afternoon is measured as it answered us, once.
  • A score is not traffic. It measures what an agent can read, not how many people arrive. Nobody can honestly convert one into the other, and we do not try.
  • The sample is well-known sites, which are better resourced than the web average. If anything these numbers are optimistic.

Citing it

AXRAY (2026-09-06). The state of Agent Experience on the public web, 2026. Dataset, 292 sites, AX specification 1.5.1. https://axray.online/research

Where would your own site sit in that distribution? One scan, about two seconds, no account.