robots.txt generator for AI crawlers

Choose a policy

Or set each group yourself. Ticked means allowed.

Answer engine 12 crawlers Indexes you so an assistant can cite and link you. Blocking it removes you from AI answers. OAI-SearchBot, Claude-SearchBot, PerplexityBot, Applebot, meta-webindexer, Amazonbot, and 6 more User-triggered 11 crawlers Fetches your page because a human asked an assistant to open it right now. Blocking it breaks a live request. ChatGPT-User, Claude-User, Perplexity-User, MistralAI-User, Google-NotebookLM, GoogleAgent-URLContext, and 5 more Autonomous agent 7 crawlers Browses, compares and buys on a person’s behalf. Blocking it removes you from the shortlist before anyone sees it. ChatGPT Agent, Operator, GoogleAgent-Mariner, Gemini-Deep-Research, AmazonBuyForMe, NovaAct, and 1 more Retrieval provider 7 crawlers Sells the index that other companies’ AI products search. Blocking it removes you from all of them at once. ExaSearchBot, TavilyBot, FirecrawlAgent, ShapBot, Diffbot, Cloudflare-AutoRAG, and 1 more Classic search 2 crawlers Traditional search indexing, increasingly the substrate AI answers are built on. Googlebot, Bingbot Coding agent 6 crawlers Reads documentation while a developer is building against you. Blocking it is how your API gets used wrong. Claude-Code, Cursor, Google-Gemini-CLI, Devin, opencode, Trae Training crawler 9 crawlers Collects content for model training. Blocking it is a legitimate business choice with no traffic cost. GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent, CCBot, and 3 more

Your domain (optional, adds the Sitemap: line)

Build my robots.txt

Your robots.txt

Copy this to https://yourdomain/robots.txt. It names every crawler you are refusing explicitly, and opens the site to everything else in one wildcard group at the end.

Copy# robots.txt - AI crawler policy
# Generated by AXRAY - https://axray.online/tools/robots-txt
#
# Allowed: answer engine, user-triggered, autonomous agent, retrieval
# provider, classic search, coding agent, training crawler.
# Refused: nothing.

# Every AI crawler in this policy is welcome, so there is nothing to
# refuse. The group below is the whole file.
# Everything not named above, including ordinary search engines and
# every AI crawler you chose to allow, may read the whole site.
User-agent: *
Allow: /

Why this asks about categories instead of bots

Because "block the AI bots" nearly always means "don't train on me", and the snippet people copy to achieve that usually also refuses the crawler that sends them visitors. The two live in the same file and look identical. OpenAI alone runs three: GPTBot collects training data, OAI-SearchBot builds the index ChatGPT cites from, and ChatGPT-User fetches your page because somebody asked for it in that moment. One line refusing all three is a decision almost nobody means to make.

So the choice here is one per category, and the file is assembled from the roster afterwards. The categories, and what each one does for you or to you:

What everyone else is doing

Measured across 292 well-known public sites on 2026-09-06. The column is how many of them refuse that crawler in robots.txt today.

Every crawler we track, one page each, with the exact user-agent string and what refusing it costs in the operator's own product.

After you publish it

1. Check it is actually served. curl -s https://yourdomain/robots.txt should print the file and nothing else. A 404 returning a styled HTML page is the commonest failure, and it means every crawler reads your site as having no policy at all.
2. Remember it is a request. Polite crawlers obey it. If you need enforcement, add a rule at your CDN or firewall as well — but leave the robots.txt in place, because a crawler that can read your policy does not have to be blocked at the edge.
3. Check nothing else is blocking you. A bot-protection product in front of the site will refuse crawlers your robots.txt just welcomed, and it does it silently. This is the failure we see most.

Questions

Should I block AI crawlers?

Some of them, and the distinction is the whole decision. Training crawlers collect text to train a model and send you nothing back, so refusing them costs you no traffic. Answer engines index you so an assistant can cite and link you, and blocking those removes you from the answers people now ask instead of searching. Most sites want the first refused and the second allowed, which is what this generator produces by default.

Does blocking GPTBot stop ChatGPT mentioning my site?

Not by itself, and this trips people up constantly. OpenAI runs more than one crawler: GPTBot collects training data, while OAI-SearchBot builds the index ChatGPT search cites from and ChatGPT-User fetches a page because a person just asked for it. Blocking GPTBot refuses training and leaves the other two working, which is usually what was meant.

Does robots.txt actually stop anyone?

It is a request, not a wall. Well-behaved crawlers from the large operators do honour it; an anonymous scraper does not, and never has. If your requirement is enforcement rather than a stated policy you need blocking at the edge — a firewall or CDN rule — and robots.txt alongside it so the polite crawlers do not have to be blocked at all.

Where does the file go?

At https://yourdomain/robots.txt, exactly there, served as plain text over HTTP 200. Not in a subdirectory, not behind a redirect chain, and not returning a 404 with a friendly HTML page, which is the failure we see most often — a site that has thought carefully about its policy and publishes it where nothing looks.

Will this break my Google ranking?

Not with the default policy: Googlebot is classic search and is allowed. Google-Extended is separate and is the one that governs training on your content for Gemini, so refusing it does not affect Search. If you tick the box that refuses classic search you will be removed from Google, and the page says so before you do it.

Is this free, and does it store anything?

Free, no account, and nothing is stored. The file is built in the request from a roster that ships with the scanner, so the tool never fetches your site at all unless you ask it for a scan afterwards.