Retrieval provider
bedrockbot is operated by Amazon. Sells the index that other companies’ AI products search. Blocking it removes you from all of them at once.
Amazon runs it as a retrieval provider, which means it crawls the web once and sells the resulting index to other companies, whose AI products search it. One crawler, many downstream products, and the name on the user agent is rarely the name of any product you have heard of.
What blocking it costs you: Enterprises building assistants on Amazon Bedrock cannot add your documentation to a knowledge base, which is exactly the use you want if you sell to developers. You leave every AI product built on that index at once, which is a great many more than the one company whose name is on the crawler.
These are two different strings and confusing them is why a rule that looks right does nothing. A robots.txt rule matches on the product token, case-insensitively, and ignores the rest of the user-agent header entirely.
| robots.txt token | bedrockbot |
|---|---|
| Operator | Amazon |
| Category | Retrieval provider |
| Full user agent | Not published by Amazon. Match on the token above. |
Where an operator has not published the exact string, this page says so rather than inventing a plausible one. A user agent you can grep your logs for is worth nothing if it is a guess.
Paste this into robots.txt at the root of your domain.
User-agent: bedrockbot
Allow: /
Sitemap: https://axray.online/sitemap.xmlAn empty Disallow: and Allow: / mean the same thing to a crawler. What does not mean the same thing is having no rule at all: absence is permission by default, but it is permission nobody wrote down, and it survives only until somebody adds a blanket User-agent: * block without thinking about this crawler.
User-agent: bedrockbot
Disallow: /Before you paste that: enterprises building assistants on Amazon Bedrock cannot add your documentation to a knowledge base, which is exactly the use you want if you sell to developers. That is the cost, and it is paid silently — no error appears in your logs, no report tells you, and the traffic you lose was traffic you never saw arrive.
Deciding one crawler at a time is how a robots.txt ends up refusing things nobody meant to refuse. The free robots.txt generator asks the question once per category instead — train on me, cite me, fetch on demand, shop on my behalf — and writes the whole file for all 54 crawlers, naming what each refusal costs before you copy it.
Measured on 2026-09-06, by reading the robots.txt of 292 public websites and resolving each one against this crawler specifically.
| Verdict | Sites | Share |
|---|---|---|
| Blocked | 8 | 3% |
| Explicitly allowed | 230 | 79% |
| No rule either way | 54 | 18% |
No site is named, here or anywhere else on this domain. The count is the useful part; a league table of businesses that never asked to be measured is not.
The row worth reading twice is the last one. 54 of 292 sites have written no rule about bedrockbot at all, which means their position on it is an accident rather than a decision — whatever their User-agent: * block happens to say.
Amazon runs 4 other crawlers with different jobs, and this is where the expensive mistake happens. Somebody means “do not train on my content”, writes one rule against the operator’s name as they remember it, and blocks the crawler that would have sent them a customer instead.
| Crawler | What it does instead |
|---|---|
Amazonbot |
it builds the index an assistant searches when somebody asks it a question. If your page is in that index you can be cited, quoted and linked in the answer. If it is not, you are not ranked lower — you are absent from the answer entirely, and no report anywhere will tell you so. |
Amzn-User |
it fetches one page at the moment a person asks an assistant to open it. There is no index and no schedule: somebody has pasted your link or asked for your site by name, and is waiting for the answer while the request happens. |
AmazonBuyForMe |
it works through a task on somebody’s behalf — compare these suppliers, book the table, find the part that fits. It reads pages to decide, and what it cannot read it cannot put on the shortlist it hands back to the person. |
NovaAct |
it works through a task on somebody’s behalf — compare these suppliers, book the table, find the part that fits. It reads pages to decide, and what it cannot read it cannot put on the shortlist it hands back to the person. |
Rules are matched per token. Blocking one of these says nothing about the others, which is the point: you can refuse training and keep every crawler that puts your link in front of a person.
Reading your own robots.txt is not the same as knowing what a crawler concludes from it. Precedence between a User-agent: * group and a named one, the longest-match rule between overlapping paths, and a token you spelled slightly wrong all produce a file that looks correct and behaves otherwise.
npx axray-cli your-site.comThe scan resolves your file against all 54 crawlers on this list individually and reports which ones are allowed, blocked, or covered by nothing. It is free, needs no account, and installs nothing.
bedrockbot is a retrieval provider operated by Amazon: it crawls the web once and sells the resulting index to other companies, whose AI products search it. One crawler, many downstream products, and the name on the user agent is rarely the name of any product you have heard of.
Only if you are willing to pay what it costs. Enterprises building assistants on Amazon Bedrock cannot add your documentation to a knowledge base, which is exactly the use you want if you sell to developers. You leave every AI product built on that index at once, which is a great many more than the one company whose name is on the crawler.
Amazon does not publish a full user-agent string for this crawler. The robots.txt product token is bedrockbot, and that is what a rule matches on; matching is case-insensitive.
Scan your site with AXRAY. It resolves your robots.txt against 54 named AI crawlers individually rather than reporting one verdict for all of them, so it will tell you whether this specific crawler is allowed, blocked, or covered by no rule at all. It is free and needs no account.
ExaSearchBot — ExaTavilyBot — TavilyFirecrawlAgent — FirecrawlShapBot — ParallelDiffbot — DiffbotCloudflare-AutoRAG — CloudflareThe full reference: all 54 AI crawlers, and who blocks each one.
See which of these 54 crawlers your own site is blocking, and what each one costs you. One scan, no account.