Cloudflare blocking AI crawlers? How to let AI search back in

Everything below comes from Cloudflare's own documentation, checked on 6 November 2026. Cloudflare renames these settings often (AI Audit became AI Crawl Control in August 2025, and the single "Block AI bots" toggle became three policies on 1 July 2026), so if a menu has moved, search the dashboard for the name in bold.

Why this matters

AI assistants can only cite pages their search crawlers can read. In our Q4 2026 Observatory, search and live-fetch agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot and others) made up about 60% of the AI fetches on a typical small-business site. A Cloudflare setting that blocks them takes you out of ChatGPT, Claude and Perplexity answers without any error you would notice.

The six settings that can block AI bots

Step 1: fix the access policies

Open Security > Settings, filter by Bot traffic and select AI Crawler and Agent access policies. There are three behaviours:

• Search: crawlers that index your content to answer questions later (OAI-SearchBot, Claude-SearchBot, PerplexityBot). Set to Allow.
• Agent: automated activity acting for a person right now, such as ChatGPT-User. Set to Allow.
• Training: crawlers that take content to train models, including mixed-purpose crawlers used for both search and training. Choose Allow or Disallow.

Disallow tells training crawlers to stay away through robots.txt and keeps the mixed-use search crawlers allowed for search. With Bot Preference Sync on, Disallow also blocks training-only crawlers at Cloudflare's edge, and mixed-use crawlers are excluded from that block. Block enforces it at Cloudflare's edge, and Cloudflare's September 2026 guidance says that stops Applebot, Bingbot and Googlebot from reaching your site, search included. That is the most damaging accidental setting we know of, because it removes you from Google, Bing and Copilot as well as from AI training.

Each behaviour also has Block on pages with ads, which blocks only on pages where Cloudflare detects advertising. If your pages carry ads and you want to be cited, leave Search and Agent on Allow.

Step 2: check AI Crawl Control

Go to AI Crawl Control and open the crawler table. Every AI crawler is listed with a category and an action:

When you block a crawler here, Cloudflare creates a custom security rule named AI Crawl Control. Cloudflare's docs warn that other custom rules running before it can still block crawlers you set to Allow, and that those blocks may not show in AI Crawl Control's numbers. So if a bot is allowed here and still failing, go to Step 4.

Step 3: Bot Fight Mode and Super Bot Fight Mode

Bot Fight Mode (Free plan) can't be skipped or given exceptions by custom rules or Page Rules, according to Cloudflare. If it challenges a crawler you want, the documented fixes are to turn it off or move to Super Bot Fight Mode. Requests it challenged show as Bot Fight Mode in the Service field under Security > Analytics > Events.

Super Bot Fight Mode (Pro and up) has three groups: Definitely automated, Likely automated and Verified bots. Cloudflare recommends Verified bots: Allow. Custom rules run before it, so you can exempt traffic with a custom rule using the Skip action.

Step 4: custom rules

Open Security > Security rules and look at every custom rule that blocks or challenges. Cloudflare warns that a Block or Managed Challenge rule can catch known bots and hurt SEO. Its documented exception for verified bots is a rule with the expression (cf.client.bot) and the action Skip for all remaining custom rules, placed first.

To exempt only AI search, AI assistants and search engines, a rule like this works on Cloudflare's verified bot categories (our construction, not a rule Cloudflare publishes):

(cf.verified_bot_category in {"AI Search" "AI Assistant" "Search Engine Crawler"})

Action Skip, placed first, above the AI Crawl Control rule. One caveat: Cloudflare keeps "AI Search" for older rules but classes new search crawlers under its July 2026 Search behaviour, so this rule can miss newer search bots. The broader (cf.client.bot) rule above covers every verified bot.

Step 5: confirm the fix

1. Reload your robots.txt. Cloudflare's managed content starts with # BEGIN Cloudflare Managed content. Cloudflare says changes appear within seconds and no cache purge is needed. Search crawlers such as OAI-SearchBot and Googlebot should not be under a Disallow: /.
2. Run the AI access checker on your domain. It reads your live robots.txt and homepage and reports, bot by bot, whether the major AI agents are allowed.
3. Watch AI Crawl Control over the next days: the crawler table shows allowed against unsuccessful requests per crawler, and the Metrics tab shows status codes per crawler.
4. Check Security > Analytics > Events for any remaining blocks, and the Service field for which feature made them.

OpenAI says OAI-SearchBot takes about 24 hours to pick up a robots.txt change, and Perplexity says changes can take up to 24 hours, so give it a day before you judge.

Before you start: what's the default?

Cloudflare's own pages disagree. Its July 2026 changelog says domains joining from 15 September 2026 have Training and Agent blocked on pages with ads, while its access-policies page (updated 2 November 2026) says new domains default to Allow for all three. Cloudflare's July 2025 announcement said new domains would be asked at setup whether to allow AI crawlers. Don't assume either way: open the policies page and read what yours says.

For which bots to allow and why, see robots.txt for AI crawlers and the AI crawler registry. If you only want to stop OpenAI training, GPTBot vs OAI-SearchBot vs ChatGPT-User shows the one line that does it.