Your bot protection is blocking the assistants that would cite you

There is a failure that does not show up in any dashboard you own. Your site is up, your uptime monitor is green, every human visitor gets a page — and an AI assistant asked about you gets a 403 and tells the person that your site could not be reached. Nothing logs this as an error, because from the edge network's point of view it did exactly what it was told.

We can put a number on how often it happens, because we hit it. When we scanned 293 well-known public websites with one plain HTTP request each, 45 of 293 (15%) refused the request outright — not a slow page, not a thin page, no page at all.

That number is a floor rather than a ceiling. It counts only outright refusals. It does not count the further 72 of 293 (25%) that answered with a consent wall, an interstitial or a login gate — pages that returned HTTP 200 and no content anybody could use.

Why this is nearly always an accident

Almost nobody sits down and decides to be absent from AI answers. What happens instead is one of four things, and all four are defaults rather than decisions.

1. A managed bot-protection rule was switched on. Most CDN and WAF products now ship a category like "AI bots" or "AI scrapers" that can be blocked with one toggle. The category is a bundle: it contains the training crawlers people mean to block and the answer engines they did not know were in there.
2. A rate limit is doing it. Assistants often fetch several pages in a burst when somebody asks a question that spans your site. A limit tuned for humans reads that as an attack.
3. A challenge page is being served. A JavaScript challenge or CAPTCHA is a 200 with no content in it. The assistant does not solve it, and what it reads instead is the challenge page — sometimes quoting that back to the person as if it were your site.
4. Someone blocked by user agent years ago. An old rule matching bot or crawler in the user-agent string catches every one of the crawlers listed on our crawler reference, including the ones that would have sent you a visitor.

How to tell whether it is happening to you

Testing this from your own browser proves nothing: you have a residential address, a real browser fingerprint and probably a session cookie. The check has to come from somewhere else, with a user agent an assistant would actually send.

Copy# What a person gets
curl -sI https://your-site.com | head -1

# What an answer engine gets
curl -sI -A "Mozilla/5.0 (compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)" \
  https://your-site.com | head -1

# What a user-triggered fetch gets, when somebody pastes your link into a chat
curl -sI -A "Mozilla/5.0 (compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)" \
  https://your-site.com | head -1

Three 200s and you are fine. Any 403, 429 or 503 in that list is the answer. A 200 is not automatically good news either — if the body that comes back is a challenge page, the status line will not tell you, and you have to look at what is in it.

Copy# A challenge page returns 200 and almost no readable text
curl -s -A "Mozilla/5.0 (compatible; OAI-SearchBot/1.0)" https://your-site.com \
  | grep -ciE "just a moment|checking your browser|enable javascript|captcha"

What to change

The fix is not to turn bot protection off. It is to stop treating one bundle of user agents as one decision, because it is two decisions with opposite consequences.

• Allow the answer engines and the user-triggered fetchers at the edge, by user agent, before any managed AI rule runs. Those are the ones that put your link in front of a person: blocking PerplexityBot or OAI-SearchBot removes you from an answer; blocking a training crawler costs you no traffic at all.
• Keep the training crawlers in whatever posture you want. That is a real business choice and this piece takes no position on it. It is simply a different lever, and it should be a different rule.
• Exempt the assistants from the human rate limit. A burst of a dozen page fetches from an answer engine is one person asking one question, not an attack.
• Never serve a JavaScript challenge to a documented crawler. If you cannot allowlist it, serve it a plain 403 instead — at least then the failure is legible to you and to them, rather than being quoted back to a customer as your website.

The exact user-agent strings for each crawler, what each one is for, and how much of the measured sample already blocks it are all on the AI crawler reference.

The part worth sitting with

These are not small sites with a misconfigured VPS. The sample is well-known companies with security teams, and being unreachable is the most complete failure on this list: every other problem is a page that gets read badly, and this one is a page that is never read at all. It is also the cheapest to fix, because it is a rule somebody switched on rather than anything about how your site is built.