The AI crawlers in your robots.txt, and the one that costs you customers

There is a distinction inside "AI crawler" that decides whether blocking one is a sensible business decision or an expensive accident, and almost nothing written about robots.txt makes it. Two crawlers from the same company can want completely different things from you.

The two kinds, and why it matters

Training crawlers collect text to train a model. Their visit brings you nothing today. Refusing them is a legitimate decision that plenty of publishers make on purpose, and nobody should tell you it is wrong.

Answer engines are different in kind. They fetch a page *at the moment a person asks a question*, so that the assistant can answer from something current and say where it came from. That fetch is the mechanism by which your name and your link appear in the answer. Blocking it does not protect your content from being learned — it removes you from the results that would have linked to you.

The two often share a company and never share a purpose. GPTBot is OpenAI's training crawler; OAI-SearchBot is what fetches for ChatGPT search. ClaudeBot and Claude-SearchBot split the same way. A rule aimed at "the AI bots" usually hits both.

Google-Extended is the one people get wrong in the other direction. It is not a crawler with its own user agent — it is a token that controls whether Google may use content it has already crawled for Gemini training. Blocking it does not remove you from Google Search.

How many sites are actually blocking these

We resolved robots.txt for 293 well-known public sites against each named crawler individually, on 2026-09-03. The interesting number is not how many block training crawlers — that is a choice — but how many block the ones that would have sent them a customer.

Read those two pairs against each other. Where the answer-engine number is close to the training number, the sites in question did not make two decisions — they made one, with a rule that caught everything.

How to write the rules deliberately

If your position is "learn from me, but only if you send people back", the file says so explicitly. Name each crawler; do not rely on a wildcard.

Copy# Answer engines: allowed, because these send people to us.
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

# Training crawlers: not allowed.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

Two things worth knowing about how these files are actually evaluated. A crawler obeys the most specific group that names it, so a User-agent: * block does not apply to GPTBot if a GPTBot group exists anywhere in the file. And an empty Disallow: means *allow everything* — it is not a typo for Disallow: /, and the difference between those two lines is the whole of your policy.

robots.txt is not the only thing that turns them away

A permissive robots.txt proves nothing on its own. Bot-protection products at the CDN layer routinely serve a challenge page to anything that is not a browser, and a crawler that receives a challenge receives no content — silently, with a 200 status, so nothing anywhere is logged as an error.

In this sample, 45 of 293 (15%) served something that looked like a wall rather than a page to a plain, honestly-identified HTTP request. Most of those sites have a robots.txt that allows the crawlers they are turning away at the edge.

What to check on your own site

1. Resolve your robots.txt against each named crawler individually rather than reading it and assuming. Precedence rules make this easy to get wrong by eye.
2. Fetch your own page with a non-browser user agent and see whether you get the page or a challenge.
3. Decide the training question deliberately, and write the answer down where the next person can see the reasoning.
4. Check it again after any CDN or bot-protection change. This is the setting that gets switched on for you by somebody who did not know what it covered.