Should You Block AI Crawlers in robots.txt?

By Idea to Traffic Editorial Team · Published October 16, 2026 · Last updated October 16, 2026

If someone ran a ChatGPT "SEO audit" on your site, there is a good chance it told you to block AI crawlers, or the opposite, to open everything up. Both answers are too blunt. There are now three different kinds of AI bots, they do different jobs, and blocking the wrong one can quietly remove you from places that send real visitors. On top of that, Cloudflare changed its defaults on September 15, 2026, so a brand new site on Cloudflare can be blocking some of these bots without the owner ever touching robots.txt.

This guide sorts the bots by job, gives you a robots.txt you can paste today, and shows how to check what your live site is actually telling crawlers.

The three kinds of AI bots

Every major AI company now runs separate user agents for separate purposes. Treat them as three groups:

• Training crawlers. They collect pages that may be used to train future models. Examples: GPTBot (OpenAI), ClaudeBot (Anthropic), and the Google-Extended token (Gemini training and grounding). Blocking these sends no visitors away today, because training does not link back to you.
• AI search crawlers. They build the index that an AI search product uses to cite and link sources. Examples: OAI-SearchBot (ChatGPT search), Claude-SearchBot, and PerplexityBot. Block these and you can disappear from answers that include clickable citations, which for some new sites is a meaningful slice of referral traffic.
• User-triggered fetchers. They load a page because a person asked the assistant to read it. Examples: ChatGPT-User and Claude-User. OpenAI says plainly that because these visits are started by a user, robots.txt rules may not apply to ChatGPT-User.

The point of separating them: you can say no to training and still say yes to being cited. For most new content sites, that is the sensible middle ground.

What blocking does (and does not do) to Google

The most common fear is that blocking AI bots will hurt Google rankings. It will not, as long as you leave Googlebot alone. Google's crawler documentation states that Google-Extended does not affect a site's inclusion in Google Search and is not used as a ranking signal. Google-Extended is only a control token in robots.txt; there is no separate bot that visits under that name.

Two things are worth knowing before you edit anything:

1. Never block Googlebot to keep pages out of AI Overviews. AI Overviews are part of Google Search, so blocking Googlebot removes you from Search entirely. If you want a specific page out of AI features, the page level snippet controls such as nosnippet are the tool, and they also shrink your normal result snippet. For a site trying to grow, that trade is rarely worth it.
2. Keep Mediapartners-Google allowed if you run AdSense. That is the AdSense crawler that reads your pages to match ads. A broad block written in a hurry, such as a User-agent: * group with Disallow: /, takes it out along with everything else.

A robots.txt you can paste today

This version blocks model training, allows AI search and citations, and leaves Google, Bing and AdSense untouched. Put it at the root of your domain as /robots.txt:

User-agent: *
Allow: /

User-agent: Mediapartners-Google
Allow: /

# Training: opt out
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /

# AI search and citations: allow
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /

Sitemap: https://example.com/sitemap.xml

Swap in your own sitemap address. If you would rather be fully open, change the three training lines to Allow: /. That is what this site does, because being read and cited by assistants is part of how a small site gets found, and the training opt out does not bring any traffic back.

A few rules that trip people up:

• A crawler obeys only the most specific group that matches its name. If you write a GPTBot group, GPTBot ignores your User-agent: * group completely, so repeat any rule you still want it to follow.
• Rules apply per host. www.example.com and blog.example.com each need their own robots.txt. Anthropic's help article calls this out for its bots specifically.
• Changes are not instant. OpenAI says it can take about 24 hours after a robots.txt update for its search systems to adjust.
• Anthropic supports the non standard Crawl-delay line, for example Crawl-delay: 1 under User-agent: ClaudeBot, if the problem is load rather than permission. Google ignores Crawl-delay.

Cloudflare's new default for pages with ads

This is the part most new site owners miss. Cloudflare's bot changelog, in an entry dated July 1, 2026, replaced the single "Block AI bots" toggle with policies by behavior: Search, Agent and Training. For new domains added from September 15, 2026, the default is that Training and Agent bots are blocked on pages that display ads, while Search stays allowed. Cloudflare also says crawlers that mix search and training are blocked by any setting that blocks AI training.

If you launched an AdSense site on Cloudflare after mid September, your ad pages may be refusing some AI agents right now even though your own robots.txt says Allow: /. That is not a disaster, since Search category bots are still allowed, but you should decide on purpose instead of by default.

To check it, open the Cloudflare dashboard, pick your domain, and go to Security, then Settings, then Configure AI bot policies. Each category offers three choices: Block on all pages, Block on pages with ads, or Allow. The old "Block AI bots" switch in the same area is marked as deprecating. If you also turned on Cloudflare's managed robots.txt, Cloudflare places its own lines in front of your file and serves them as one, so the file a bot sees is not the same as the file in your repository.

Check what your live site actually says

Do not trust the file in your project folder. Fetch the live one:

curl -s https://example.com/robots.txt

If you see lines you did not write, that is Cloudflare's managed robots.txt. Then test how the server responds to each bot by sending its user agent:

for ua in GPTBot OAI-SearchBot ClaudeBot Claude-SearchBot PerplexityBot Googlebot; do
  printf "%-18s " "$ua"
  curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 (compatible; $ua)" https://example.com/
done

A 200 means the page was served. A 403 means a firewall or bot rule is blocking that agent before robots.txt even matters. Keep in mind that Cloudflare verifies real bots by more than the user agent string, so a spoofed curl request is a quick signal, not proof. The definitive view is Security, then Analytics in Cloudflare, filtered by bot category, or your server logs.

While you are at it, confirm the basics did not break. Our free Site Checker reads your live robots.txt and flags a blocked Googlebot, a missing sitemap line and similar structural problems in about ten seconds.

Which choice fits your site

• New site, under about 1,000 visits a month: allow AI search bots. You need every source of discovery. Blocking training is a values choice, not an SEO one, so pick whichever you are comfortable with.
• Ad funded site with original data or tools: block training, allow search. Your value is in the page itself, and citations with links are better than having your numbers absorbed into an answer without credit.
• Server struggling under crawler load: look at logs first. Unverified scrapers that pretend to be AI bots often ignore robots.txt anyway, and a Cloudflare rule or rate limit handles them better than a Disallow line.

Whatever you choose, write it down in robots.txt explicitly. A short, deliberate file beats a default you never looked at, and it is the first thing the next audit, human or AI, will read.

Where this question came from

• r/SEO: Question about AI crawlers and robots.txt
• Cloudflare: Block AI Bots documentation
• OpenAI: Overview of OpenAI crawlers

Questions about blocking AI crawlers