Coding agent
Cursor is operated by Cursor. Reads documentation while a developer is building against you. Blocking it is how your API gets used wrong.
Cursor runs it as a coding agent, which means it reads documentation while a developer is building something against your API, usually with the developer watching. What it cannot fetch, the developer gets as a guess from the model instead.
What blocking it costs you: The editor a large share of developers now write in cannot pull up your reference page while they are building against you. A developer building against your API reads whatever they can get instead of your documentation, and integrates against a guess.
These are two different strings and confusing them is why a rule that looks right does nothing.
A robots.txt rule matches on the product token, case-insensitively, and
ignores the rest of the user-agent header entirely.
| robots.txt token | Cursor |
|---|---|
| Operator | Cursor |
| Category | Coding agent |
| Full user agent | Not published by Cursor. Match on the token above. |
Where an operator has not published the exact string, this page says so rather than inventing a plausible one. A user agent you can grep your logs for is worth nothing if it is a guess.
Paste this into robots.txt at the root of your domain.
User-agent: Cursor
Allow: /
Sitemap: https://axray.online/sitemap.xml
An empty Disallow: and Allow: / mean the same thing to a crawler. What does
not mean the same thing is having no rule at all: absence is permission by default, but it is
permission nobody wrote down, and it survives only until somebody adds a blanket
User-agent: * block without thinking about this crawler.
User-agent: Cursor
Disallow: /Before you paste that: the editor a large share of developers now write in cannot pull up your reference page while they are building against you. That is the cost, and it is paid silently — no error appears in your logs, no report tells you, and the traffic you lose was traffic you never saw arrive.
Measured on 2026-09-06, by reading the robots.txt of
292 public websites and resolving each one against this crawler specifically.
| Verdict | Sites | Share |
|---|---|---|
| Blocked | 6 | 2% |
| Explicitly allowed | 232 | 79% |
| No rule either way | 54 | 18% |
No site is named, here or anywhere else on this domain. The count is the useful part; a league table of businesses that never asked to be measured is not.
The row worth reading twice is the last one. 54 of 292 sites have written no rule about
Cursor at all, which means their position on it is an accident rather than a decision — whatever
their User-agent: * block happens to say.
Reading your own robots.txt is not the same as knowing what a crawler concludes from it.
Precedence between a User-agent: * group and a named one, the longest-match rule between
overlapping paths, and a token you spelled slightly wrong all produce a file that looks correct and
behaves otherwise.
npx axray-cli your-site.comThe scan resolves your file against all 54 crawlers on this list individually and reports which ones are allowed, blocked, or covered by nothing. It is free, needs no account, and installs nothing.
Cursor is a coding agent operated by Cursor: it reads documentation while a developer is building something against your API, usually with the developer watching. What it cannot fetch, the developer gets as a guess from the model instead.
Only if you are willing to pay what it costs. The editor a large share of developers now write in cannot pull up your reference page while they are building against you. A developer building against your API reads whatever they can get instead of your documentation, and integrates against a guess.
Cursor does not publish a full user-agent string for this crawler. The robots.txt product token is Cursor, and that is what a rule matches on; matching is case-insensitive.
Scan your site with AXRAY. It resolves your robots.txt against 54 named AI crawlers individually rather than reporting one verdict for all of them, so it will tell you whether this specific crawler is allowed, blocked, or covered by no rule at all. It is free and needs no account.
Claude-Code — AnthropicGoogle-Gemini-CLI — GoogleDevin — Cognitionopencode — opencodeTrae — ByteDanceThe full reference: all 54 AI crawlers, and who blocks each one.
See which of these 54 crawlers your own site is blocking, and what each one costs you. One scan, no account.