User-triggered
ChatGPT-User is operated by OpenAI. Fetches your page because a human asked an assistant to open it right now. Blocking it breaks a live request.
OpenAI runs it as a user-triggered fetcher, which means it fetches one page at the moment a person asks an assistant to open it. There is no index and no schedule: somebody has pasted your link or asked for your site by name, and is waiting for the answer while the request happens.
What blocking it costs you: When a user asks ChatGPT to open your link, it fails in front of them. A link somebody pasted into an assistant fails while they are watching. They asked for your page by name and got an error.
These are two different strings and confusing them is why a rule that looks right does nothing.
A robots.txt rule matches on the product token, case-insensitively, and
ignores the rest of the user-agent header entirely.
| robots.txt token | ChatGPT-User |
|---|---|
| Operator | OpenAI |
| Category | User-triggered |
| Full user agent | Mozilla/5.0 (compatible; ChatGPT-User/1.0; +https://openai.com/bot) |
Paste this into robots.txt at the root of your domain.
User-agent: ChatGPT-User
Allow: /
Sitemap: https://axray.online/sitemap.xml
An empty Disallow: and Allow: / mean the same thing to a crawler. What does
not mean the same thing is having no rule at all: absence is permission by default, but it is
permission nobody wrote down, and it survives only until somebody adds a blanket
User-agent: * block without thinking about this crawler.
User-agent: ChatGPT-User
Disallow: /Before you paste that: when a user asks ChatGPT to open your link, it fails in front of them. That is the cost, and it is paid silently — no error appears in your logs, no report tells you, and the traffic you lose was traffic you never saw arrive.
Measured on 2026-09-06, by reading the robots.txt of
292 public websites and resolving each one against this crawler specifically.
| Verdict | Sites | Share |
|---|---|---|
| Blocked | 27 | 9% |
| Explicitly allowed | 211 | 72% |
| No rule either way | 54 | 18% |
No site is named, here or anywhere else on this domain. The count is the useful part; a league table of businesses that never asked to be measured is not.
The row worth reading twice is the last one. 54 of 292 sites have written no rule about
ChatGPT-User at all, which means their position on it is an accident rather than a decision — whatever
their User-agent: * block happens to say.
OpenAI runs 4 other crawlers with different jobs, and this is where the expensive mistake happens. Somebody means “do not train on my content”, writes one rule against the operator’s name as they remember it, and blocks the crawler that would have sent them a customer instead.
| Crawler | What it does instead |
|---|---|
OAI-SearchBot |
Indexes you so an assistant can cite and link you. Blocking it removes you from AI answers. |
ChatGPT Agent |
Browses, compares and buys on a person’s behalf. Blocking it removes you from the shortlist before anyone sees it. |
Operator |
Browses, compares and buys on a person’s behalf. Blocking it removes you from the shortlist before anyone sees it. |
GPTBot |
Collects content for model training. Blocking it is a legitimate business choice with no traffic cost. |
Rules are matched per token. Blocking one of these says nothing about the others, which is the point: you can refuse training and keep every crawler that puts your link in front of a person.
Reading your own robots.txt is not the same as knowing what a crawler concludes from it.
Precedence between a User-agent: * group and a named one, the longest-match rule between
overlapping paths, and a token you spelled slightly wrong all produce a file that looks correct and
behaves otherwise.
npx axray-cli your-site.comThe scan resolves your file against all 54 crawlers on this list individually and reports which ones are allowed, blocked, or covered by nothing. It is free, needs no account, and installs nothing.
ChatGPT-User is a user-triggered fetcher operated by OpenAI: it fetches one page at the moment a person asks an assistant to open it. There is no index and no schedule: somebody has pasted your link or asked for your site by name, and is waiting for the answer while the request happens.
Only if you are willing to pay what it costs. When a user asks ChatGPT to open your link, it fails in front of them. A link somebody pasted into an assistant fails while they are watching. They asked for your page by name and got an error.
Mozilla/5.0 (compatible; ChatGPT-User/1.0; +https://openai.com/bot) — and the robots.txt product token, which is what a rule actually matches on, is ChatGPT-User.
Scan your site with AXRAY. It resolves your robots.txt against 54 named AI crawlers individually rather than reporting one verdict for all of them, so it will tell you whether this specific crawler is allowed, blocked, or covered by no rule at all. It is free and needs no account.
Claude-User — AnthropicPerplexity-User — PerplexityMistralAI-User — MistralGoogle-NotebookLM — GoogleGoogleAgent-URLContext — Googlekagi-fetcher — Kagimeta-externalfetcher — MetaKimi-User — Moonshot AIThe full reference: all 54 AI crawlers, and who blocks each one.
See which of these 54 crawlers your own site is blocking, and what each one costs you. One scan, no account.