Reachability · robots-providers
A handful of companies sell the index that most AI products search rather than crawling themselves. Blocking one of these removes you from every application built on it at once, and none of those applications will ever appear in your logs.
| Check id | robots-providers |
|---|---|
| Pillar | Reachability — Can an agent fetch your content at all, without a browser and without being blocked? |
| Weight within the pillar | 5 points |
| Share of the overall score | about 1.0 of 100 |
| Kind of change | Engineering — a code change and a deploy |
Two numbers rather than one, because a check's weight is inside its pillar and the pillar is a share of the whole. The largest check in the smallest pillar moves your score less than a middling one in the largest, and knowing which you are looking at is the difference between fixing the right thing and fixing the loudest thing.
A handful of companies sell the index that most AI products search rather than crawling themselves. Blocking one of these removes you from every application built on it at once, and none of those applications will ever appear in your logs.
Worth saying plainly: this is not a search-engine ranking factor and passing it will not move you up a results page. It measures something else — whether software with no browser, no JavaScript and a few hundred tokens of budget can tell what your page is and do something with it. Those are the conditions an AI assistant reads you under, and they are much harsher than a browser's.
This check does not carry a copy-pasteable snippet, and the reason is worth stating rather than papering over: what it measures is a property of your server or your whole document, not a tag that can be pasted into it. A handful of companies sell the index that most AI products search rather than crawling themselves. Blocking one of these removes you from every application built on it at once, and none of those applications will ever appear in your logs.
Scan the page and the report will say what was actually observed — the header that was missing, the status that came back, the passage that could not stand alone — which is the part that tells you what to change. A generic snippet here would be a guess about a site nobody has looked at.
Measured on 2026-09-06 against AX specification v1.5.1, across a sample of 292 public websites.
| Result | Sites | Share of those assessed |
|---|---|---|
| Failed | 11 | 4% |
| Warned | 25 | 9% |
| Passed | 256 | 88% |
| Did not apply | 0 | excluded from the shares |
The last row is kept out of the percentages deliberately. A check that did not apply to 0 of 292 sites is not a check most of the web passes, and folding those in is the commonest way this sort of table is used to say something untrue.
At 4% this is something most sites already get right, which makes failing it more costly than the point value suggests — you are behind the field rather than level with it.
npx axray-cli your-site.com
All 70 checks against one URL, this one among them, with what was found rather than only whether it
passed. Add --min 70 in a pipeline and the command exits non-zero below that score, so a
change that breaks what agents can read fails the build instead of shipping.
A handful of companies sell the index that most AI products search rather than crawling themselves. Blocking one of these removes you from every application built on it at once, and none of those applications will ever appear in your logs. It sits in the Reachability pillar of the Agent Experience rubric and is worth 5 points within that pillar, which works out at about 1.0 points of the overall score out of 100.
No. Of 292 public sites where this check applied, 11 failed it — 4%. The measurement was taken on 2026-09-06 against AX specification v1.5.1, and no site is named.
Run npx axray-cli your-site.com from a terminal. It runs all 70 checks against one URL and prints this one among them, with what it found rather than only whether it passed. It is free, needs no account and installs nothing.
Can an agent fetch your content at all, without a browser and without being blocked?
See how your own page does on this check and the other 69, with the fixes ordered by how many points each one recovers. One scan, no account.