Can an AI assistant read my site at all?

Fetch your own page with no browser and read what comes back. Most AI crawlers do not run JavaScript, so a page assembled in the visitor’s browser arrives as an empty shell. This is the first thing to establish, because everything else is downstream of it.

Which AI crawlers am I blocking, and what does each one cost me?

A crawler that trains on you and a crawler that cites you are different decisions, and one robots.txt line usually makes both. AXRAY resolves robots.txt against 54 named crawlers individually, and there is a page for each saying what blocking it costs.

What has to be true before an assistant cites me?

Four things in order: the crawler is allowed in, your content arrives without a browser, a passage of a few hundred tokens stands alone without the rest of the page, and that passage says something specific enough to quote. Most sites lose at the first or second.

Guides to AI visibility, crawlers and checks

Where should I start?

Three questions in the order they matter, each with the page that answers it.

Can an AI assistant read my site at all? — the first thing to establish, because everything else is downstream of it.
Which AI crawlers am I blocking, and what does each one cost me? — the decision most sites make by accident.
What has to be true before an assistant cites me? — four things, in order, and most sites lose at the first.

What is worth reading?

Measured pieces rather than opinion: every number in them is regenerated from the repository.

How to get cited by ChatGPT, Claude and Perplexity — Four things have to happen before an assistant names you as a source. Three are under your control, one is not, and most sites fail at the first.
Generative Engine Optimization, and how much of it is just SEO — GEO means being readable and citable by AI assistants rather than ranked by a search engine. What genuinely differs from SEO, and what is the same work renamed.
Nobody reads your page. They read about 300 tokens of it. — An assistant is handed one passage of your page, a few hundred tokens long, and answers from that alone. What that does to writing meant to be read from the top.
llms.txt vs robots.txt vs sitemap.xml: which does what — Three text files at your domain root, constantly confused. One grants permission, one lists every URL, one introduces the site — and only two are read today.
How to check whether an AI assistant can read your website — Six checks you can run yourself with curl, in about five minutes, that between them catch almost everything that makes a site unreadable to an AI assistant.
Your bot protection is blocking the assistants that would cite you — Of 292 well-known websites we fetched, 45 refused a plain HTTP request outright. Almost none of them meant to.
Why your React site is nearly invisible to ChatGPT — Crawling assistants fetch HTML and do not run JavaScript. If your content is assembled in the browser, an empty shell arrives — and nothing reports an error.
The AI crawlers in your robots.txt, and the one that costs you customers — Some AI crawlers train on your content. Others fetch you the moment a customer asks a question, and cite you. Most sites block both with one rule.
llms.txt: what it is, and whether anyone is actually using it — A plain-text map of your site written for AI assistants, not browsers. The format, what it does and does not buy you, and how many real sites publish one.
The state of Agent Experience, measured on 292 sites — We scanned 292 well-known websites the way an AI assistant reads them. The median scores 68 out of 100, and the failures are not the ones anyone expects.

The full index · Türkçe yazılar

Which AI crawlers should I block?

54 of them, each with its robots.txt token, the exact user agent, what blocking it costs in the operator’s own product, and how many of 292 measured sites already block it.

Answer engine

Indexes you so an assistant can cite and link you. Blocking it removes you from AI answers.

User-triggered

Fetches your page because a human asked an assistant to open it right now. Blocking it breaks a live request.

Autonomous agent

Browses, compares and buys on a person’s behalf. Blocking it removes you from the shortlist before anyone sees it.

Training crawler

Collects content for model training. Blocking it is a legitimate business choice with no traffic cost.

All 54, including retrieval providers, coding agents and classic search

What does each check measure?

All 70 checks in the Agent Experience rubric, grouped by pillar. Each says what it measures, what it is worth, how to satisfy it, and how often the web fails it.

Reachability — 19 checks, weight 25. Can an agent fetch your content at all, without a browser and without being blocked?
Page responds to a plain HTTP request · No bot wall in front of the content · Content exists without running JavaScript · The status code matches what the page says
Comprehension — 16 checks, weight 25. Once fetched, can the agent work out what this page is and what it says?
Has a descriptive <title> · Has a meta description · The page describes itself consistently · Exactly one <h1> naming the page
Structure — 15 checks, weight 18. Is the important data machine-readable instead of implied by visual layout?
Ships structured data · Structured data parses cleanly · Structured data describes the right thing · Structured data agrees with the page
Actionability — 11 checks, weight 17. Can the agent take the next step - navigate, search, submit, buy - without guessing?
Navigation uses real links · Link text describes the destination · Links onward into the site · Form fields are labelled
Agent Contract — 9 checks, weight 15. Have you told agents the rules: what they may use, where the docs are, who to contact?
Publishes an llms.txt · llms.txt is actually useful · States an explicit AI policy · Declares AI usage preferences the standard way

The full list, all 70 of them

What do these words mean?

The glossary defines the eight terms this subject is argued in, including the three that are the same thing under different names.

What can I run right now?

The llms.txt generator — paste a URL, get a first draft built from your own site’s name, description and navigation. Free, no account, and it does not invent the descriptions.
The scanner — paste a URL, get a score out of 100 and an ordered fix list. One free scan a day with no account.
The command line tool — npx axray-cli example.com, free, nothing to install, and it can fail a build below a threshold.
The AX Index — sites whose owners verified their domain and published their score.
Compare two sites — the same rubric applied to both, side by side.

Or skip the reading and see where your own site stands. About two seconds, no account.

Everything here, arranged by what you came to find out