The Agent Experience specification

Why publish a specification instead of just a tool?

A score is only worth something if you can argue with it. Naming the pillars and their weights means you can see what we value and hold us to the same standard we apply to you - our own site is scored by this engine like everyone else's.

The rubric is versioned. Scores are comparable within a major version and we will not silently reweight underneath you. Reports record the spec version they were produced against.

How is the score calculated?

• Each check earns points toward its pillar: full for a pass, half for a warning, zero for a failure.
• Checks marked not applicable are removed from the denominator entirely.
• Pillar scores are 0–100 within the pillar, then combined by the weights below.
• Gates cap the total when a failure makes the average misleading.

Every check, one page each

All 70 of them, grouped by pillar, with what each is worth inside that pillar. These pages are generated from the scoring code, so a check cannot say one thing here and score another thing on your report. The index ranks the same list by how many sites in a measured sample fail each one.

Reachability

• Page responds to a plain HTTP request — 10 points
• No bot wall in front of the content — 12 points
• Content exists without running JavaScript — 14 points
• The status code matches what the page says — 5 points
• robots.txt is present and parseable — 4 points, configuration only
• Answer engines are allowed to crawl — 12 points
• User-triggered fetchers are allowed — 8 points
• Autonomous agents are allowed — 8 points
• Retrieval providers are allowed — 5 points
• Classic search crawlers are allowed — 6 points
• Coding agents can read the documentation — 4 points
• Real agent user-agents are served normally — 8 points
• No directive suppressing AI use of this page — 6 points
• Short redirect chain — 4 points
• Responds fast enough for an agent budget — 5 points
• HTML payload stays inside an agent context budget — 5 points
• HTML is compressed on the wire — 3 points
• Re-fetching is cheap for a crawler — 4 points, configuration only
• Declares a correct content type — 3 points, configuration only

Comprehension

• Has a descriptive <title> — 10 points, configuration only
• Has a meta description — 8 points, configuration only
• The page describes itself consistently — 6 points
• Exactly one <h1> naming the page — 8 points, configuration only
• Headings form a sane outline — 6 points, configuration only
• Enough substance to be worth citing — 10 points
• Answers questions in a form that can be quoted — 8 points
• Passages make sense on their own — 7 points
• Signal-to-markup ratio — 8 points
• Declares a document language — 5 points, configuration only
• Text arrived intact, not mangled — 5 points
• Images carry alt text — 8 points
• Main content is marked as such — 6 points, configuration only
• Uses semantic elements, not div soup — 6 points
• Provides a no-JavaScript fallback — 4 points
• URL describes the content — 3 points

Structure

• Ships structured data — 12 points, configuration only
• Structured data parses cleanly — 8 points
• Structured data describes the right thing — 8 points
• Structured data agrees with the page — 7 points
• Open Graph card is complete — 6 points, configuration only
• Declares a canonical URL — 6 points, configuration only
• Publishes a sitemap — 8 points, configuration only
• Publishes a feed of its content — 4 points
• Declares its language alternates — 4 points
• Prices are machine-readable — 6 points
• Tabular data is marked up as a table — 4 points
• Dates are machine-readable — 3 points
• Declares its place in the site — 4 points
• Declares a link preview card — 3 points, configuration only
• Inline microdata annotations — 2 points

Actionability

• Navigation uses real links — 12 points
• Link text describes the destination — 8 points
• Links onward into the site — 6 points
• Form fields are labelled — 8 points
• Form fields have stable names — 6 points
• Fields declare autocomplete tokens — 4 points
• Controls are real buttons — 6 points
• Reachable by a human, discoverably — 5 points, configuration only
• Site search is machine-usable — 4 points, configuration only
• Declares what an agent can do here — 5 points, configuration only
• Programmatic surface is discoverable — 6 points, configuration only

Agent Contract

• Publishes an llms.txt — 14 points, configuration only
• llms.txt is actually useful — 6 points
• States an explicit AI policy — 6 points, configuration only
• Declares AI usage preferences the standard way — 5 points, configuration only
• Exposes an agent-callable manifest — 6 points
• Identity is machine-verifiable — 5 points, configuration only
• Says when the content changed — 5 points, configuration only
• Declares content licensing — 5 points, configuration only
• Publishes security.txt — 4 points, configuration only

When does a single failure cap the whole score?

Some failures make a good average untrue, so they cap the total outright rather than averaging in: a page that never returns successfully, a bot wall answering in its place, a robots.txt that turns away answer engines and search alike, a directive that suppresses indexing or snippets, and a page that ships no content without JavaScript. When a cap is what decided your score, your report names it and explains why, because an unexplained ceiling reads as a bug and an explained one reads as a finding.

Can the rubric be gamed?

It used to be, and closing that is ongoing work. The rubric is public and stays public: a scanner selling a standard cannot hide the standard, and a score you are not allowed to argue with is not worth having. But publishing a rubric also publishes a recipe for satisfying it: put an empty file at /llms.txt, emit a JSON-LD block with a type and no properties, add an h1 that repeats your domain name, label a form field with its own name attribute. Every one of those used to score full marks here, and none of them changes anything for an agent.

So every check that could be satisfied by an artefact merely existing now looks at what is in it. An llms.txt has to say something and link somewhere. A Product has to carry offers; an Article has to carry datePublished. A security.txt has to name a contact. A manifest has to name an endpoint. A title, an h1 and a meta description have to be three different sentences rather than one sentence three times. Word counts are measured against the page's own content rather than the navigation and footer that repeat site-wide.

Where the artefact is present but hollow, the check keeps its status, earns at most half its points, and is labelled thin evidence on your report with a sentence saying what is missing. That is deliberate: a silent deduction reads as a bug, and a check that quietly failed you for something you did on purpose would be worse than one that explains itself.

We publish this because saying it out loud is itself the deterrent, and because a rubric that can be gamed and pretends otherwise is the thing actually worth distrusting. If you find a way to score well here without being more useful to an agent, tell us and we will close it — that is a bug in the rubric, and the rubric is the product.

What score earns which grade?

A+ 95+ · A 85–94 · B 72–84 · C 58–71 · D 44–57 · E 28–43 · F below 28

Which crawlers is robots.txt resolved against?

robots.txt is resolved for each of these crawlers individually, grouped by what blocking them costs you.

Answer engine

Indexes you so an assistant can cite and link you. Blocking it removes you from AI answers.

• OAI-SearchBot — OpenAI. You will not appear as a source in ChatGPT search results.
• Claude-SearchBot — Anthropic. You will not be cited in Claude web search answers.
• PerplexityBot — Perplexity. You lose Perplexity citations, one of the highest-converting AI referral sources.
• Applebot — Apple. Siri, Spotlight and the answers inside Apple Intelligence all read Apple’s own index. Blocking Applebot removes you from every one of them.
• meta-webindexer — Meta. Meta AI stops citing and linking you inside WhatsApp, Instagram and Facebook, where a very large number of people now ask their questions.
• Amazonbot — Amazon. Excluded from Alexa and Rufus style shopping answers.
• DuckAssistBot — DuckDuckGo. Excluded from DuckDuckGo AI assist answers.
• Bravebot — Brave. You leave the Brave index, which answers in Brave’s own assistant and is also licensed to a long list of other AI products that never crawl you themselves.
• YouBot — You.com. You.com’s search and its assistant stop returning you as a source.
• TongyiBot — Alibaba. Qwen and the Tongyi assistant answer a very large Chinese-speaking audience without you in the sources.
• PetalBot — Huawei. Huawei’s search and assistant, which are the default on a large installed base of devices, stop recommending you.
• YandexAdditional — Yandex. YandexGPT stops using you for the quick answers shown above ordinary Yandex results.

User-triggered

Fetches your page because a human asked an assistant to open it right now. Blocking it breaks a live request.

• ChatGPT-User — OpenAI. When a user asks ChatGPT to open your link, it fails in front of them.
• Claude-User — Anthropic. Claude cannot open your page when a user pastes the link.
• Perplexity-User — Perplexity. Live user-requested fetches from Perplexity fail.
• MistralAI-User — Mistral. Le Chat cannot open your page for the person who just asked it to.
• Google-NotebookLM — Google. Somebody adding your page as a source in NotebookLM gets an empty source. This is the one fetch that only ever happens because a person deliberately chose you.
• GoogleAgent-URLContext — Google. A developer who passes your URL to the Gemini API as context gets nothing back, and usually blames their own code rather than your robots.txt.
• kagi-fetcher — Kagi. Kagi’s assistant and summariser cannot read a page its subscriber asked for.
• meta-externalfetcher — Meta. Meta AI cannot open a link a person shared with it in a chat.
• Kimi-User — Moonshot AI. Kimi cannot summarise your page when a reader asks it to.
• Amzn-User — Amazon. Alexa and the other Amazon assistants cannot fetch your page to answer a question somebody just asked out loud.
• cohere-ai — Cohere. Cohere’s grounded generation, which sits behind a large number of enterprise assistants, cannot retrieve you in response to a prompt.

Autonomous agent

Browses, compares and buys on a person’s behalf. Blocking it removes you from the shortlist before anyone sees it.

• ChatGPT Agent — OpenAI. The agent that browses, compares and fills in forms on a customer’s behalf cannot get past your front door, so you are never on the shortlist it comes back with.
• Operator — OpenAI. OpenAI’s browsing agent cannot complete a booking, a signup or a checkout on your site.
• GoogleAgent-Mariner — Google. Google’s browsing agent cannot carry out a multi-step task on your site.
• Gemini-Deep-Research — Google. Deep Research reports — the long, cited documents people increasingly make decisions from — are written without you in them.
• AmazonBuyForMe — Amazon. The agent that buys a product from a merchant’s own site on a customer’s behalf cannot read your product pages, so it buys somewhere else.
• NovaAct — Amazon. Amazon’s browser agent cannot operate your site on a user’s behalf.
• Manus-User — Butterfly Effect. Manus, a general task agent with a large user base, cannot use your site as part of a job somebody asked it to finish.

Retrieval provider

Sells the index that other companies’ AI products search. Blocking it removes you from all of them at once.

• ExaSearchBot — Exa. Exa is the retrieval layer behind a large number of AI products that never crawl the web themselves. Blocking it removes you from all of them in one line.
• TavilyBot — Tavily. Tavily is the default search tool in a great many agent frameworks. If it cannot see you, neither can anything built on it.
• FirecrawlAgent — Firecrawl. Firecrawl is how a large share of AI applications turn a page into text. A block here is a block on every one of them.
• ShapBot — Parallel. Parallel’s research and extraction API, used to ground agents that need accurate citations, stops returning you.
• Diffbot — Diffbot. You disappear from a knowledge graph that a great many companies buy rather than build, which is often where an assistant gets what it believes about your company.
• Cloudflare-AutoRAG — Cloudflare. Anyone building a retrieval application on Cloudflare cannot include your pages in it.
• bedrockbot — Amazon. Enterprises building assistants on Amazon Bedrock cannot add your documentation to a knowledge base, which is exactly the use you want if you sell to developers.

Classic search

Traditional search indexing, increasingly the substrate AI answers are built on.

• Googlebot — Google. You are invisible to Google Search and to AI Overviews built on it.
• Bingbot — Microsoft. You are invisible to Bing and to Copilot answers grounded on it.

Coding agent

Reads documentation while a developer is building against you. Blocking it is how your API gets used wrong.

• Claude-Code — Anthropic. A developer integrating with you cannot have the agent read your documentation, so it writes the integration from memory and gets your API wrong.
• Cursor — Cursor. The editor a large share of developers now write in cannot pull up your reference page while they are building against you.
• Google-Gemini-CLI — Google. Gemini CLI cannot read your docs while a developer works.
• Devin — Cognition. An autonomous engineering agent cannot consult your documentation mid-task.
• opencode — opencode. An open-source terminal coding agent cannot read your reference material.
• Trae — ByteDance. ByteDance’s coding agent cannot fetch your documentation while generating code against it.

Training crawler

Collects content for model training. Blocking it is a legitimate business choice with no traffic cost.

• GPTBot — OpenAI. Your content is excluded from OpenAI model training.
• ClaudeBot — Anthropic. Your content is excluded from Anthropic model training.
• Google-Extended — Google. Excluded from Gemini training and grounding.
• Applebot-Extended — Apple. Excluded from Apple Intelligence training.
• meta-externalagent — Meta. Excluded from Meta AI training.
• CCBot — Common Crawl. Excluded from Common Crawl, the base corpus behind most open models.
• Bytespider — ByteDance. Excluded from ByteDance model training. This is the crawler most often reported for ignoring the rule, so verify at the edge rather than trusting robots.txt alone.
• DeepSeekBot — DeepSeek. Excluded from DeepSeek model training.
• AI2Bot — Ai2. Excluded from the open training corpora published by the Allen Institute, which are what most academic and open-weight models are built from.