guide

How to block AI crawlers on your own site

A number of AI companies run automated crawlers that fetch pages to train models, to answer search queries, or to fetch a page on a user's behalf. Each one announces itself with a User-Agent string, documented by its own operator. Below are copyable rules to block the announced list at your server, your CDN, or in robots.txt.

Two things worth knowing before you use any of this. A User-Agent header is whatever the sender chose to send, so a rule matches what a request claimed to be, not who actually sent it; a rule here can be worked around by anything willing to lie about its identity. And robots.txt is a request, not a lock: a crawler that honours it stops, and nothing stops the rest. Want to see which of these, if any, actually reach your site before you block anything? Try the free log checker.

The list this covers

Tokens below are the announced crawler user agents, read from each operator's own documentation: OpenAI (GPTBot, OAI-SearchBot, ChatGPT-User); Anthropic (ClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, claude-web); Common Crawl (CCBot); Perplexity (PerplexityBot, Perplexity-User); ByteDance (Bytespider); Amazon (Amazonbot); Meta (Meta-ExternalAgent, Meta-ExternalFetcher, FacebookBot); Apple (Applebot-Extended); Google (Google-Extended); Cohere (cohere-ai, cohere-training-data-crawler); Mistral AI (MistralAI-User); DuckDuckGo (DuckAssistBot); You.com (YouBot); Allen Institute for AI (AI2Bot); Diffbot (Diffbot); Timpi (Timpibot); Webz.io (omgili); ImageSift (ImagesiftBot). A user agent can be forged, so this says what a request claims to be, not who sent it. Source list and citations: src/aicrawlers.js in this site's repository.

nginx

Inside the server { } block for your site. Reload nginx after adding it (nginx -t, then nginx -s reload). Matched requests get a 403.

# Inside the server { } block for your site
if ($http_user_agent ~* "(GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|anthropic-ai|claude-web|CCBot|PerplexityBot|Perplexity-User|Bytespider|Amazonbot|Meta-ExternalAgent|Meta-ExternalFetcher|FacebookBot|cohere-ai|cohere-training-data-crawler|MistralAI-User|DuckAssistBot|YouBot|AI2Bot|Diffbot|Timpibot|omgili|ImagesiftBot)") {
    return 403;
}

Apache

In the virtual host, or in .htaccess when AllowOverride permits rewrites. Needs mod_rewrite. Matched requests get a 403.

# In the virtual host or .htaccess (needs mod_rewrite)
<IfModule mod_rewrite.c>
    RewriteEngine On
    RewriteCond %{HTTP_USER_AGENT} (GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|anthropic-ai|claude-web|CCBot|PerplexityBot|Perplexity-User|Bytespider|Amazonbot|Meta-ExternalAgent|Meta-ExternalFetcher|FacebookBot|cohere-ai|cohere-training-data-crawler|MistralAI-User|DuckAssistBot|YouBot|AI2Bot|Diffbot|Timpibot|omgili|ImagesiftBot) [NC]
    RewriteRule ^ - [F,L]
</IfModule>

Cloudflare (WAF rule)

Security → WAF → Custom rules → Create rule → Edit expression: paste this and choose Block.

(lower(http.user_agent) contains "gptbot") or (lower(http.user_agent) contains "oai-searchbot") or (lower(http.user_agent) contains "chatgpt-user") or (lower(http.user_agent) contains "claudebot") or (lower(http.user_agent) contains "claude-searchbot") or (lower(http.user_agent) contains "claude-user") or (lower(http.user_agent) contains "anthropic-ai") or (lower(http.user_agent) contains "claude-web") or (lower(http.user_agent) contains "ccbot") or (lower(http.user_agent) contains "perplexitybot") or (lower(http.user_agent) contains "perplexity-user") or (lower(http.user_agent) contains "bytespider") or (lower(http.user_agent) contains "amazonbot") or (lower(http.user_agent) contains "meta-externalagent") or (lower(http.user_agent) contains "meta-externalfetcher") or (lower(http.user_agent) contains "facebookbot") or (lower(http.user_agent) contains "cohere-ai") or (lower(http.user_agent) contains "cohere-training-data-crawler") or (lower(http.user_agent) contains "mistralai-user") or (lower(http.user_agent) contains "duckassistbot") or (lower(http.user_agent) contains "youbot") or (lower(http.user_agent) contains "ai2bot") or (lower(http.user_agent) contains "diffbot") or (lower(http.user_agent) contains "timpibot") or (lower(http.user_agent) contains "omgili") or (lower(http.user_agent) contains "imagesiftbot")

robots.txt

Put this at the root of your site as /robots.txt. It only asks; see the note above on which crawlers actually stop for it.

# robots.txt is a request, not a lock: crawlers that honour it stop, others do not.
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: anthropic-ai
User-agent: claude-web
User-agent: CCBot
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Bytespider
User-agent: Amazonbot
User-agent: Meta-ExternalAgent
User-agent: Meta-ExternalFetcher
User-agent: FacebookBot
User-agent: cohere-ai
User-agent: cohere-training-data-crawler
User-agent: MistralAI-User
User-agent: DuckAssistBot
User-agent: YouBot
User-agent: AI2Bot
User-agent: Diffbot
User-agent: Timpibot
User-agent: omgili
User-agent: ImagesiftBot
User-agent: Applebot-Extended
User-agent: Google-Extended
Disallow: /

Check it worked

A rule you cannot verify is a guess. Paste your access log into the free log checker to see which of these, if any, hit your site before and after you apply a rule; the log is read in your browser and never uploaded. If you would rather not do this yourself, Signal Nodus applies rules like these to your site and checks they hold for 350 USD: the done-for-you service.