Answer engines
These read you so an assistant can cite you and link you. Blocking one removes you from that assistant’s answers entirely - not "less visible", absent.
Reference
Which AI crawlers should you actually block?
Every crawler below is one this scanner probes with, so the list is the product rather than a copy of somebody else’s. The last column is measured: it is how many sites in a sample of 292 public websites already block that crawler in robots.txt.
The distinction that decides everything
User-triggered fetchers
These fetch a page because a person asked an assistant to open it, right now, with the person waiting. Blocking one means the link somebody pasted fails in front of them.
Autonomous agents
These browse, compare and complete a task on somebody’s behalf: shortlist three suppliers, book the table, buy the part. Blocking one does not make you look worse than the competition. It removes you from the comparison before the customer ever sees it.
Retrieval providers
A handful of companies crawl the web once and sell the index that everybody else’s AI product searches. Blocking one of these takes you out of every application built on it at the same moment, and not one of those applications will ever show up in your logs.
Classic search crawlers
Ordinary search indexing, which increasingly is also the substrate AI answers are built on. Almost nobody means to block these.
Coding agents
These read documentation while a developer is building against you. Blocking one does not stop the integration; it makes the agent write it from memory, and the wrong version of your API is what ships.
Training crawlers
These collect content to train models. Blocking them is a legitimate business decision and it costs you no traffic, which is exactly what makes it the safe one to take.
The full list
| Crawler | Operator | What it is for | Already blocked by |
|---|---|---|---|
OAI-SearchBot |
OpenAI | Answer engines | 20 of 292 |
Claude-SearchBot |
Anthropic | Answer engines | 26 of 292 |
PerplexityBot |
Perplexity | Answer engines | 36 of 292 |
Applebot |
Apple | Answer engines | 5 of 292 |
meta-webindexer |
Meta | Answer engines | 12 of 292 |
Amazonbot |
Amazon | Answer engines | 35 of 292 |
DuckAssistBot |
DuckDuckGo | Answer engines | 20 of 292 |
Bravebot |
Brave | Answer engines | 9 of 292 |
YouBot |
You.com | Answer engines | 26 of 292 |
TongyiBot |
Alibaba | Answer engines | 8 of 292 |
PetalBot |
Huawei | Answer engines | 26 of 292 |
YandexAdditional |
Yandex | Answer engines | 8 of 292 |
ChatGPT-User |
OpenAI | User-triggered fetchers | 27 of 292 |
Claude-User |
Anthropic | User-triggered fetchers | 27 of 292 |
Perplexity-User |
Perplexity | User-triggered fetchers | 23 of 292 |
MistralAI-User |
Mistral | User-triggered fetchers | 21 of 292 |
Google-NotebookLM |
User-triggered fetchers | 9 of 292 | |
GoogleAgent-URLContext |
User-triggered fetchers | 6 of 292 | |
kagi-fetcher |
Kagi | User-triggered fetchers | 7 of 292 |
meta-externalfetcher |
Meta | User-triggered fetchers | 20 of 292 |
Kimi-User |
Moonshot AI | User-triggered fetchers | 7 of 292 |
Amzn-User |
Amazon | User-triggered fetchers | 7 of 292 |
cohere-ai |
Cohere | User-triggered fetchers | 37 of 292 |
ChatGPT Agent |
OpenAI | Autonomous agents | 6 of 292 |
Operator |
OpenAI | Autonomous agents | 6 of 292 |
GoogleAgent-Mariner |
Autonomous agents | 8 of 292 | |
Gemini-Deep-Research |
Autonomous agents | 9 of 292 | |
AmazonBuyForMe |
Amazon | Autonomous agents | 8 of 292 |
NovaAct |
Amazon | Autonomous agents | 9 of 292 |
Manus-User |
Butterfly Effect | Autonomous agents | 10 of 292 |
ExaSearchBot |
Exa | Retrieval providers | 6 of 292 |
TavilyBot |
Tavily | Retrieval providers | 10 of 292 |
FirecrawlAgent |
Firecrawl | Retrieval providers | 11 of 292 |
ShapBot |
Parallel | Retrieval providers | 12 of 292 |
Diffbot |
Diffbot | Retrieval providers | 36 of 292 |
Cloudflare-AutoRAG |
Cloudflare | Retrieval providers | 8 of 292 |
bedrockbot |
Amazon | Retrieval providers | 8 of 292 |
Googlebot |
Classic search crawlers | 0 of 292 | |
Bingbot |
Microsoft | Classic search crawlers | 1 of 292 |
Claude-Code |
Anthropic | Coding agents | 7 of 292 |
Cursor |
Cursor | Coding agents | 6 of 292 |
Google-Gemini-CLI |
Coding agents | 5 of 292 | |
Devin |
Cognition | Coding agents | 11 of 292 |
opencode |
opencode | Coding agents | 6 of 292 |
Trae |
ByteDance | Coding agents | 6 of 292 |
GPTBot |
OpenAI | Training crawlers | 41 of 292 |
ClaudeBot |
Anthropic | Training crawlers | 43 of 292 |
Google-Extended |
Training crawlers | 37 of 292 | |
Applebot-Extended |
Apple | Training crawlers | 37 of 292 |
meta-externalagent |
Meta | Training crawlers | 34 of 292 |
CCBot |
Common Crawl | Training crawlers | 55 of 292 |
Bytespider |
ByteDance | Training crawlers | 46 of 292 |
DeepSeekBot |
DeepSeek | Training crawlers | 16 of 292 |
AI2Bot |
Ai2 | Training crawlers | 18 of 292 |
Blocking figures measured on 2026-09-06 across 292 public websites. No site is named.
Each one, and what blocking it costs
OAI-SearchBot
OpenAI · Answer engines
Blocking it means: You will not appear as a source in ChatGPT search results.
User agent: Mozilla/5.0 (compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)
20 of 292 · 54 sites have no robots.txt rule either way.
Claude-SearchBot
Anthropic · Answer engines
Blocking it means: You will not be cited in Claude web search answers.
User agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; +Claude-SearchBot@anthropic.com)
26 of 292 · 54 sites have no robots.txt rule either way.
PerplexityBot
Perplexity · Answer engines
Blocking it means: You lose Perplexity citations, one of the highest-converting AI referral sources.
User agent: Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)
36 of 292 · 54 sites have no robots.txt rule either way.
Applebot
Apple · Answer engines
Blocking it means: Siri, Spotlight and the answers inside Apple Intelligence all read Apple’s own index. Blocking Applebot removes you from every one of them.
User agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot)
5 of 292 · 54 sites have no robots.txt rule either way.
meta-webindexer
Meta · Answer engines
Blocking it means: Meta AI stops citing and linking you inside WhatsApp, Instagram and Facebook, where a very large number of people now ask their questions.
12 of 292 · 54 sites have no robots.txt rule either way.
Amazonbot
Amazon · Answer engines
Blocking it means: Excluded from Alexa and Rufus style shopping answers.
User agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36 (compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)
35 of 292 · 54 sites have no robots.txt rule either way.
DuckAssistBot
DuckDuckGo · Answer engines
Blocking it means: Excluded from DuckDuckGo AI assist answers.
User agent: Mozilla/5.0 (compatible; DuckAssistBot/1.0; +https://duckduckgo.com/duckassistbot/)
20 of 292 · 54 sites have no robots.txt rule either way.
Bravebot
Brave · Answer engines
Blocking it means: You leave the Brave index, which answers in Brave’s own assistant and is also licensed to a long list of other AI products that never crawl you themselves.
9 of 292 · 54 sites have no robots.txt rule either way.
YouBot
You.com · Answer engines
Blocking it means: You.com’s search and its assistant stop returning you as a source.
User agent: Mozilla/5.0 (compatible; YouBot (+http://www.you.com))
26 of 292 · 54 sites have no robots.txt rule either way.
TongyiBot
Alibaba · Answer engines
Blocking it means: Qwen and the Tongyi assistant answer a very large Chinese-speaking audience without you in the sources.
8 of 292 · 54 sites have no robots.txt rule either way.
PetalBot
Huawei · Answer engines
Blocking it means: Huawei’s search and assistant, which are the default on a large installed base of devices, stop recommending you.
26 of 292 · 54 sites have no robots.txt rule either way.
YandexAdditional
Yandex · Answer engines
Blocking it means: YandexGPT stops using you for the quick answers shown above ordinary Yandex results.
8 of 292 · 54 sites have no robots.txt rule either way.
ChatGPT-User
OpenAI · User-triggered fetchers
Blocking it means: When a user asks ChatGPT to open your link, it fails in front of them.
User agent: Mozilla/5.0 (compatible; ChatGPT-User/1.0; +https://openai.com/bot)
27 of 292 · 54 sites have no robots.txt rule either way.
Claude-User
Anthropic · User-triggered fetchers
Blocking it means: Claude cannot open your page when a user pastes the link.
User agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +Claude-User@anthropic.com)
27 of 292 · 54 sites have no robots.txt rule either way.
Perplexity-User
Perplexity · User-triggered fetchers
Blocking it means: Live user-requested fetches from Perplexity fail.
User agent: Mozilla/5.0 (compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)
23 of 292 · 54 sites have no robots.txt rule either way.
MistralAI-User
Mistral · User-triggered fetchers
Blocking it means: Le Chat cannot open your page for the person who just asked it to.
21 of 292 · 54 sites have no robots.txt rule either way.
Google-NotebookLM
Google · User-triggered fetchers
Blocking it means: Somebody adding your page as a source in NotebookLM gets an empty source. This is the one fetch that only ever happens because a person deliberately chose you.
9 of 292 · 54 sites have no robots.txt rule either way.
GoogleAgent-URLContext
Google · User-triggered fetchers
Blocking it means: A developer who passes your URL to the Gemini API as context gets nothing back, and usually blames their own code rather than your robots.txt.
6 of 292 · 54 sites have no robots.txt rule either way.
kagi-fetcher
Kagi · User-triggered fetchers
Blocking it means: Kagi’s assistant and summariser cannot read a page its subscriber asked for.
7 of 292 · 54 sites have no robots.txt rule either way.
meta-externalfetcher
Meta · User-triggered fetchers
Blocking it means: Meta AI cannot open a link a person shared with it in a chat.
20 of 292 · 54 sites have no robots.txt rule either way.
Kimi-User
Moonshot AI · User-triggered fetchers
Blocking it means: Kimi cannot summarise your page when a reader asks it to.
7 of 292 · 54 sites have no robots.txt rule either way.
Amzn-User
Amazon · User-triggered fetchers
Blocking it means: Alexa and the other Amazon assistants cannot fetch your page to answer a question somebody just asked out loud.
7 of 292 · 54 sites have no robots.txt rule either way.
cohere-ai
Cohere · User-triggered fetchers
Blocking it means: Cohere’s grounded generation, which sits behind a large number of enterprise assistants, cannot retrieve you in response to a prompt.
37 of 292 · 54 sites have no robots.txt rule either way.
ChatGPT Agent
OpenAI · Autonomous agents
Blocking it means: The agent that browses, compares and fills in forms on a customer’s behalf cannot get past your front door, so you are never on the shortlist it comes back with.
6 of 292 · 54 sites have no robots.txt rule either way.
Operator
OpenAI · Autonomous agents
Blocking it means: OpenAI’s browsing agent cannot complete a booking, a signup or a checkout on your site.
6 of 292 · 54 sites have no robots.txt rule either way.
GoogleAgent-Mariner
Google · Autonomous agents
Blocking it means: Google’s browsing agent cannot carry out a multi-step task on your site.
8 of 292 · 54 sites have no robots.txt rule either way.
Gemini-Deep-Research
Google · Autonomous agents
Blocking it means: Deep Research reports — the long, cited documents people increasingly make decisions from — are written without you in them.
9 of 292 · 54 sites have no robots.txt rule either way.
AmazonBuyForMe
Amazon · Autonomous agents
Blocking it means: The agent that buys a product from a merchant’s own site on a customer’s behalf cannot read your product pages, so it buys somewhere else.
8 of 292 · 54 sites have no robots.txt rule either way.
NovaAct
Amazon · Autonomous agents
Blocking it means: Amazon’s browser agent cannot operate your site on a user’s behalf.
9 of 292 · 54 sites have no robots.txt rule either way.
Manus-User
Butterfly Effect · Autonomous agents
Blocking it means: Manus, a general task agent with a large user base, cannot use your site as part of a job somebody asked it to finish.
10 of 292 · 54 sites have no robots.txt rule either way.
ExaSearchBot
Exa · Retrieval providers
Blocking it means: Exa is the retrieval layer behind a large number of AI products that never crawl the web themselves. Blocking it removes you from all of them in one line.
6 of 292 · 54 sites have no robots.txt rule either way.
TavilyBot
Tavily · Retrieval providers
Blocking it means: Tavily is the default search tool in a great many agent frameworks. If it cannot see you, neither can anything built on it.
10 of 292 · 54 sites have no robots.txt rule either way.
FirecrawlAgent
Firecrawl · Retrieval providers
Blocking it means: Firecrawl is how a large share of AI applications turn a page into text. A block here is a block on every one of them.
11 of 292 · 54 sites have no robots.txt rule either way.
ShapBot
Parallel · Retrieval providers
Blocking it means: Parallel’s research and extraction API, used to ground agents that need accurate citations, stops returning you.
12 of 292 · 54 sites have no robots.txt rule either way.
Diffbot
Diffbot · Retrieval providers
Blocking it means: You disappear from a knowledge graph that a great many companies buy rather than build, which is often where an assistant gets what it believes about your company.
36 of 292 · 54 sites have no robots.txt rule either way.
Cloudflare-AutoRAG
Cloudflare · Retrieval providers
Blocking it means: Anyone building a retrieval application on Cloudflare cannot include your pages in it.
8 of 292 · 54 sites have no robots.txt rule either way.
bedrockbot
Amazon · Retrieval providers
Blocking it means: Enterprises building assistants on Amazon Bedrock cannot add your documentation to a knowledge base, which is exactly the use you want if you sell to developers.
8 of 292 · 54 sites have no robots.txt rule either way.
Googlebot
Google · Classic search crawlers
Blocking it means: You are invisible to Google Search and to AI Overviews built on it.
User agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
0 of 292 · 54 sites have no robots.txt rule either way.
Bingbot
Microsoft · Classic search crawlers
Blocking it means: You are invisible to Bing and to Copilot answers grounded on it.
User agent: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
1 of 292 · 54 sites have no robots.txt rule either way.
Claude-Code
Anthropic · Coding agents
Blocking it means: A developer integrating with you cannot have the agent read your documentation, so it writes the integration from memory and gets your API wrong.
7 of 292 · 54 sites have no robots.txt rule either way.
Cursor
Cursor · Coding agents
Blocking it means: The editor a large share of developers now write in cannot pull up your reference page while they are building against you.
6 of 292 · 54 sites have no robots.txt rule either way.
Google-Gemini-CLI
Google · Coding agents
Blocking it means: Gemini CLI cannot read your docs while a developer works.
5 of 292 · 54 sites have no robots.txt rule either way.
Devin
Cognition · Coding agents
Blocking it means: An autonomous engineering agent cannot consult your documentation mid-task.
11 of 292 · 54 sites have no robots.txt rule either way.
opencode
opencode · Coding agents
Blocking it means: An open-source terminal coding agent cannot read your reference material.
6 of 292 · 54 sites have no robots.txt rule either way.
Trae
ByteDance · Coding agents
Blocking it means: ByteDance’s coding agent cannot fetch your documentation while generating code against it.
6 of 292 · 54 sites have no robots.txt rule either way.
GPTBot
OpenAI · Training crawlers
Blocking it means: Your content is excluded from OpenAI model training.
User agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)
41 of 292 · 54 sites have no robots.txt rule either way.
ClaudeBot
Anthropic · Training crawlers
Blocking it means: Your content is excluded from Anthropic model training.
User agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)
43 of 292 · 54 sites have no robots.txt rule either way.
Google-Extended
Google · Training crawlers
Blocking it means: Excluded from Gemini training and grounding.
User agent: Mozilla/5.0 (compatible; Google-Extended/1.0)
37 of 292 · 54 sites have no robots.txt rule either way.
Applebot-Extended
Apple · Training crawlers
Blocking it means: Excluded from Apple Intelligence training.
User agent: Mozilla/5.0 (compatible; Applebot-Extended/1.0)
37 of 292 · 54 sites have no robots.txt rule either way.
meta-externalagent
Meta · Training crawlers
Blocking it means: Excluded from Meta AI training.
User agent: meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)
34 of 292 · 54 sites have no robots.txt rule either way.
CCBot
Common Crawl · Training crawlers
Blocking it means: Excluded from Common Crawl, the base corpus behind most open models.
User agent: CCBot/2.0 (https://commoncrawl.org/faq/)
55 of 292 · 54 sites have no robots.txt rule either way.
Bytespider
ByteDance · Training crawlers
Blocking it means: Excluded from ByteDance model training. This is the crawler most often reported for ignoring the rule, so verify at the edge rather than trusting robots.txt alone.
46 of 292 · 54 sites have no robots.txt rule either way.
DeepSeekBot
DeepSeek · Training crawlers
Blocking it means: Excluded from DeepSeek model training.
16 of 292 · 54 sites have no robots.txt rule either way.
AI2Bot
Ai2 · Training crawlers
Blocking it means: Excluded from the open training corpora published by the Allen Institute, which are what most academic and open-weight models are built from.
18 of 292 · 54 sites have no robots.txt rule either way.
A robots.txt that separates the two decisions
This is the shape worth copying: refuse training, keep every crawler that could send you a visitor. Change the last group if you want to be trained on - that is the only line in it that is a matter of opinion.
# Everything that could send you a visitor. Keep these.
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Applebot
User-agent: meta-webindexer
User-agent: Amazonbot
User-agent: DuckAssistBot
User-agent: Bravebot
User-agent: YouBot
User-agent: TongyiBot
User-agent: PetalBot
User-agent: YandexAdditional
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
User-agent: MistralAI-User
User-agent: Google-NotebookLM
User-agent: GoogleAgent-URLContext
User-agent: kagi-fetcher
User-agent: meta-externalfetcher
User-agent: Kimi-User
User-agent: Amzn-User
User-agent: cohere-ai
User-agent: ChatGPT Agent
User-agent: Operator
User-agent: GoogleAgent-Mariner
User-agent: Gemini-Deep-Research
User-agent: AmazonBuyForMe
User-agent: NovaAct
User-agent: Manus-User
User-agent: ExaSearchBot
User-agent: TavilyBot
User-agent: FirecrawlAgent
User-agent: ShapBot
User-agent: Diffbot
User-agent: Cloudflare-AutoRAG
User-agent: bedrockbot
User-agent: Googlebot
User-agent: Bingbot
User-agent: Claude-Code
User-agent: Cursor
User-agent: Google-Gemini-CLI
User-agent: Devin
User-agent: opencode
User-agent: Trae
Allow: /
# Training crawlers. This is the only group that is a matter of opinion.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: CCBot
User-agent: Bytespider
User-agent: DeepSeekBot
User-agent: AI2Bot
Disallow: /
Sitemap: https://axray.online/sitemap.xmlOne rule that people get wrong: a crawler matches the most specific user-agent group that names it, and nothing else. A crawler named in its own group ignores your User-agent: * block entirely, including the Disallow lines you wanted to keep.
You do not have to assemble this by hand: the free robots.txt generator asks the question once per category rather than once per crawler — train on me, cite me, fetch on demand, shop on my behalf — and writes the whole file, naming what each refusal costs.
Questions people actually ask
Does blocking GPTBot stop ChatGPT from citing my site?
No. GPTBot collects content for training. The crawler that puts you into ChatGPT answers is OAI-SearchBot, and the one that opens a link when somebody asks ChatGPT to read it is ChatGPT-User. Blocking all three is a common accident, because most published robots.txt advice lists them together as "the OpenAI bots".
Is robots.txt legally binding on AI crawlers?
It is a request, not a fence, and it has never been anything else - the same was true of search crawlers for twenty-five years. The major operators listed here document the tokens they honour. If you need enforcement rather than a request, that is a firewall rule or a licence agreement, not a text file.
What happens if I have no robots.txt at all?
Everything is allowed, which is the default and is usually fine. In the sample behind this page, 54 of 292 sites had no rule either way for the main training crawlers. The risk of an absent file is not that you are crawled; it is that you have made no decision and cannot tell anyone what your decision was.
Will blocking AI crawlers protect my content from being used?
Only by the operators that honour robots.txt, which is the well-behaved subset. It does nothing about content that reaches a model through a third-party copy, a screenshot, or a user pasting your text into a chat window. Blocking is worth doing as a stated position; it is not a technical guarantee and nobody should sell it to you as one.
See which of these your own robots.txt currently blocks, and what each one costs you - in one scan, with no account.