Blocking AI crawlers properly: robots.txt, nginx, Apache and Cloudflare

There are two different jobs here and they get confused constantly. Stating a policy is what robots.txt does: it tells well-behaved crawlers what you permit, and the large operators do honour it. Enforcing a policy needs a rule at the edge — your web server or your CDN — because an anonymous scraper reads your robots.txt exactly as attentively as it chooses to.

Do both, in that order, and understand which one you are doing. A site that blocks at the edge but publishes no robots.txt forces every polite crawler to be refused the hard way, and tells nobody why.

Layer one: state the policy

This is the one to get right first, because it is the one the crawlers you *want* also read. The important thing is not the syntax — it is that "AI crawler" is not one thing. OpenAI alone runs three: GPTBot collects training data, OAI-SearchBot builds the index ChatGPT cites from, and ChatGPT-User fetches your page because somebody asked for it in that moment. One rule refusing all three is a decision almost nobody means to make.

Copy# Refuse training. Keep everything that can send a visitor.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /

User-agent: *
Allow: /

The free generator writes the complete version of that file, naming every crawler we track and printing what each refusal costs before you copy it. Every crawler has its own page with its exact token and operator.

A crawler matches the single most specific User-agent group that names it and ignores every other group. A bot named in its own group therefore ignores your User-agent: * block entirely — including the Disallow lines you wanted to keep. This is the commonest way a robots.txt does the opposite of what it looks like.

Layer two: enforce it at the edge

If your requirement is that a crawler genuinely cannot read the page, the rule has to sit in front of the application. All three configurations below match on the user-agent string, which is what a polite crawler sends and a determined scraper forges — so this raises the cost of taking your content, and does not make it impossible.

nginx

Copymap $http_user_agent $ai_training_bot {
    default              0;
    "~*GPTBot"           1;
    "~*ClaudeBot"        1;
    "~*Google-Extended"  1;
    "~*CCBot"            1;
    "~*Bytespider"       1;
}

server {
    # ...
    if ($ai_training_bot) {
        return 403;
    }
}

Use map rather than a chain of if blocks: if inside a server block is evaluated per request and is notoriously awkward about what it can safely contain, while map is resolved once at config load.

Apache

Copy<IfModule mod_setenvif.c>
  SetEnvIfNoCase User-Agent "GPTBot"          ai_training
  SetEnvIfNoCase User-Agent "ClaudeBot"       ai_training
  SetEnvIfNoCase User-Agent "Google-Extended" ai_training
  SetEnvIfNoCase User-Agent "CCBot"           ai_training

  <RequireAll>
    Require all granted
    Require not env ai_training
  </RequireAll>
</IfModule>

Cloudflare

A WAF custom rule, expression editor, action Block:

Copy(http.user_agent contains "GPTBot") or
(http.user_agent contains "ClaudeBot") or
(http.user_agent contains "Google-Extended") or
(http.user_agent contains "CCBot")

Cloudflare also ships a one-click "Block AI Scrapers and Crawlers" toggle. It is convenient and it is blunt: read its current crawler list before enabling it, because what counts as an AI scraper to a vendor and what counts as one to you are different questions, and the toggle does not ask.

The mistake this article is really about

Bot protection you did not configure for this purpose is already refusing crawlers you want. We scanned 292 public sites and 72 of 292 (25%) of them answer an ordinary agent request with an interstitial, a challenge or a block page instead of their content. A further 45 refuse outright.

Almost none of those are deliberate. A managed WAF rule, a bot-fight mode, a rate limiter tuned for scrapers — each one is doing its job, and the assistant that was about to cite you gets a challenge page. Nothing appears in your logs as an error. The traffic you lose is traffic you never saw arrive.

So the order of work is: decide the policy, publish it in robots.txt, enforce the part you actually need at the edge — and then check what an agent really receives, because the layer you did not touch is usually the one refusing.

Copynpx axray-cli your-site.com --probe

--probe fetches your origin with each real crawler user-agent in turn, so it reports what each one is actually served rather than what your configuration says it should be. Why bot protection blocks AI assistants is the longer version of this failure.

What blocking does not achieve

• It does not remove you from a model that already trained on you. These rules apply from now on.
• It does not stop text reaching a model another way — a syndicated copy, a screenshot, somebody pasting your page into a chat window.
• It does not stop a scraper that forges its user-agent. Every method here matches on a string the client chooses to send.
• It does not affect Google Search. Googlebot and Google-Extended are separate: the first is search, the second governs training. Refusing the second does not affect your ranking.