robots.txt for AI crawlers: what blocking each one costs you

robots.txt is a request, not a lock. The well-documented crawlers honour it; a few say they may not for user-triggered fetches, and some publish nothing at all. It's still the main switch you have, and in our Observatory data the AI agents that obey it were most of the traffic. Every token below links back to its operator's docs in the crawler registry.

What blocking each crawler costs

Sources: each operator's crawler documentation, linked in the registry. Checked October 2026.

Recipe 1: allow everything (the default)

If you have no robots.txt, every crawler is already allowed. If you want one, this is all it needs. The Sitemap line helps every crawler find your pages.

User-agent: *
Allow: /

Sitemap: https://yourbusiness.com/sitemap.xml

Recipe 2: opt out of AI training, stay in AI search

This is the version we'd suggest to a business that's uneasy about training but wants to be recommended. Several User-agent lines can share one group (RFC 9309 allows it), so the training bots get one Disallow together. Everything else, including every search and live-fetch agent, falls through to the * group and stays allowed.

# AI training crawlers and opt-out tokens
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Bytespider
User-agent: MistralAI-Training
User-agent: cohere-ai
Disallow: /

# Everyone else, including AI search and Google
User-agent: *
Allow: /

Sitemap: https://yourbusiness.com/sitemap.xml

Add User-agent: Amazonbot to the first group if you also want out of Amazon's general crawl. Amazon says Alexa's search and live answers use Amzn-SearchBot and Amzn-User, which this recipe leaves allowed.

Recipe 3: keep a private area out of every crawler

Robots.txt rules are path prefixes, and the longest match wins. This keeps a client portal or staging folder out of every bot while the rest of the site stays open. Don't rely on it for anything secret; anyone can read robots.txt, and it lists the paths you're hiding. Use a login for that.

User-agent: *
Disallow: /portal/
Disallow: /drafts/
Allow: /

Sitemap: https://yourbusiness.com/sitemap.xml

Recipe 4: slow a crawler down instead of blocking it

If a crawler is hammering a small server, Crawl-delay asks for a pause between requests. Anthropic and Common Crawl document support for it. Amazon says it doesn't support Crawl-delay, and Google doesn't either, so don't add it for them.

User-agent: ClaudeBot
User-agent: CCBot
Crawl-delay: 5

User-agent: *
Allow: /

Mistakes we see most

• One "block AI" switch that blocks search too. Squarespace's "Artificial Intelligence Crawlers" setting, for example, disallows 26 agents, including DuckAssistBot and YouBot, which feed AI answers rather than training (per this Squarespace designer's guide, updated August 2025, which cites Squarespace's support page). Check what your platform's toggle writes to robots.txt.
• Blocking Googlebot to avoid AI Overviews. Google says Googlebot rules are the control for Search, so this removes you from Google entirely. Use nosnippet, max-snippet or data-nosnippet if you want Google to show less of your text.
• Blocking ChatGPT-User and assuming it worked. OpenAI says robots.txt rules may not apply to user-initiated visits. If you need that block, it has to happen at the firewall, and it will stop ChatGPT reading you for customers who ask.
• A robots.txt that errors. Under RFC 9309, crawlers should treat a robots.txt that returns a 5xx error as "disallow everything". A missing file (404) is fine; a broken one isn't.
• Expecting instant effect. OpenAI and Perplexity say changes take about 24 hours, DuckDuckGo says 72 hours, and Amazon may use a cached copy up to 30 days old.

Check your result

Paste your domain into the AI access checker after any change. It reads your live robots.txt and shows each of the 21 agents and tokens as allowed or blocked, with the rule that decided it.