AI Crawler List 2026: User Agents, robots.txt Tokens and IP Verification

How to use this

There are three kinds of AI agent, and they cost you very different things when you block them. Search agents (OAI-SearchBot, Claude-SearchBot, PerplexityBot, DuckAssistBot) are how assistants find you to cite you, so keep them open. Training agents (GPTBot, ClaudeBot, CCBot) only affect whether your pages train future models, so blocking them is a reasonable choice with no direct visibility cost. User-triggered agents (ChatGPT-User, Claude-User, Perplexity-User) fetch a page because a person just asked about it, and several of them say robots.txt may not apply.

Every user agent, robots.txt token and verification method below was checked against the operator's own documentation in October 2026; each row links to it. Where an operator publishes nothing (ByteDance and Cohere, for example), we say so instead of repeating what other lists guess. The "In our data" column is the mean share of a site's AI fetches across the 12 small-business sites in our Q4 2026 Observatory.

26 crawlers and tokens

Bot and user agent	Purpose	What blocking it costs	Obeys robots.txt	Verify	In our data
GPTBotOpenAICrawls pages that may be used to train OpenAI's foundation models.`Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot`	Model training	Your future pages stay out of OpenAI training data. OpenAI says this does not affect ChatGPT search, which uses OAI-SearchBot.	Yes	Published IP list · Docs	10.1%
OAI-SearchBotOpenAIBuilds the index that ChatGPT search uses to surface and link to websites.`Mozilla/5.0 … compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot (version may change)`	AI search index	You drop out of ChatGPT search results and their citations. This is the one to keep open if you want ChatGPT to link to you.	Yes (changes take about 24 hours)	Published IP list · Docs	14.9%
ChatGPT-UserOpenAIFetches a page live when a ChatGPT user or a Custom GPT asks about it.`Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot`	User-triggered fetch	ChatGPT can't read your page when someone asks about you directly. OpenAI says it does not decide whether you appear in search.	May not apply: OpenAI says robots.txt rules may not apply to user-initiated visits	Published IP list · Docs	28.1%
ClaudeBotAnthropicCollects public web content that may be used to train Anthropic's models.`Contains “ClaudeBot” (Anthropic does not publish the full string)`	Model training	Your future pages are excluded from Anthropic training datasets. Claude's search and live fetches use separate agents.	Yes, including the non-standard Crawl-delay	Published IP list · Docs	14.7%
Claude-SearchBotAnthropicCrawls to improve the quality of Claude's search results.`Contains “Claude-SearchBot”`	AI search index	Anthropic says your content won't be indexed for Claude's search, which may reduce your visibility in its answers.	Yes	Published IP list · Docs	1.4%
Claude-UserAnthropicFetches a page when a Claude user's question needs it.`Contains “Claude-User”`	User-triggered fetch	Claude can't read your page for a user who asks, which Anthropic says may reduce your visibility in user-directed search.	Yes (Anthropic says its bots honour robots.txt)	Published IP list · Docs	4.2%
PerplexityBotPerplexitySurfaces and links websites in Perplexity's search results. Perplexity says it is not used to train foundation models.`Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)`	AI search index	You stop appearing as a cited source in Perplexity answers.	Yes (changes take up to 24 hours)	Published IP list · Docs	6.9%
Perplexity-UserPerplexityVisits a page when a Perplexity user's question needs it, and links to it in the answer.`Mozilla/5.0 (compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)`	User-triggered fetch	A robots.txt block is unlikely to stop it. Blocking at the firewall stops Perplexity reading you live for a user.	Generally ignores robots.txt, because a user asked for the fetch	Published IP list · Docs	2.1%
GooglebotGoogleGoogle Search's crawler. AI Overviews and AI Mode are built from the same index, so this is also Google's AI-answer crawler.`Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)`	AI search index	You disappear from Google Search, AI Overviews and AI Mode. Never block it to avoid AI: use nosnippet or max-snippet instead.	Yes	Reverse DNS to googlebot.com, or Google's IP list · Docs	–
Google-ExtendedGoogleNot a crawler. A robots.txt token that controls whether content Google already crawls may be used for Gemini training and grounding.`None: Google says it has no separate HTTP user agent`	robots.txt token only	Google says it does not affect inclusion or ranking in Google Search. It limits use of your content by Gemini and other Google AI systems.	It is a robots.txt rule	Nothing to verify: any request claiming to be Google-Extended is not from Google · Docs	–
Google-CloudVertexBotGoogleCrawls sites whose owners asked for it while building Vertex AI agents.`Contains “Google-CloudVertexBot”`	User-triggered fetch	Google says it has no effect on Google Search or other products.	Yes (also obeys Googlebot rules)	Google's crawler IP ranges · Docs	0.3%
ApplebotApplePowers search in Spotlight, Siri and Safari. Apple says crawled data may also help train its foundation models and ground AI answers.`Safari-style string ending (Applebot/0.1; +http://www.apple.com/go/applebot)`	Mixed use	You drop out of Siri, Spotlight and Safari suggestions. To opt out of training only, block Applebot-Extended instead.	Yes	Reverse DNS to applebot.apple.com, or Apple's IP list · Docs	–
Applebot-ExtendedAppleNot a crawler. A robots.txt token that controls whether Applebot's data trains Apple's foundation models.`None: Apple says Applebot-Extended does not crawl webpages`	robots.txt token only	Apple says it does not affect search ranking. It only opts you out of foundation-model training.	It is a robots.txt rule	Nothing to verify · Docs	–
BingbotMicrosoftBing's crawler. Microsoft Copilot's web answers are grounded in the Bing index, and there is no separate Copilot token.`Contains “bingbot/2.0; +http://www.bing.com/bingbot.htm”`	AI search index	You lose Bing, Copilot, and the engines that use Bing's index. Rarely worth it.	Yes	Reverse DNS to search.msn.com, or Bing's IP list · Docs	–
meta-externalagentMetaCrawls for uses such as training Meta's foundation AI models or indexing content directly.`meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)`	Model training	Keeps your pages out of Meta AI training. Link previews on Facebook and Instagram use a different agent (facebookexternalhit).	Meta tells you to block it with a robots.txt Disallow	Not documented by Meta · Docs	7.2%
meta-externalfetcherMetaFetches individual links when a user asks, for Meta's agentic AI features.`meta-externalfetcher/1.1`	User-triggered fetch	Robots.txt may not stop it. Low volume in our data.	May bypass robots.txt, per Meta	Not documented by Meta · Docs	–
AmazonbotAmazonUsed to improve Amazon products and services. Amazon says it may be used to train Amazon AI models.`Contains “compatible; Amazonbot/0.1”`	Mixed use	Keeps you out of Amazonbot's crawl, including possible training. Alexa search and live answers use Amzn-SearchBot and Amzn-User.	Yes (no Crawl-delay; may use a cached robots.txt up to 30 days old)	Published IP list (on the docs page) · Docs	–
Amzn-SearchBotAmazonImproves search experiences such as Alexa. Amazon says it does not gather content for generative AI training.`Contains “compatible; Amzn-SearchBot/0.1”`	AI search index	You may drop out of Alexa answers that rely on web search.	Yes	Published IP list (on the docs page) · Docs	–
Amzn-UserAmazonFetches live web information when an Alexa user asks something that needs it.`Contains “compatible; Amzn-User/0.1”`	User-triggered fetch	Alexa can't read your page live for a user.	May not follow all directives (user-initiated)	Published IP list (on the docs page) · Docs	–
BytespiderByteDanceByteDance's crawler. ByteDance publishes no crawler documentation we could find; it is widely described as collecting data for its AI models.`Contains “Bytespider”`	Model training	Little known upside to allowing it. If robots.txt isn't respected, a firewall rule is the only reliable block.	Not documented by ByteDance	No published method · No official docs	4.5%
CCBotCommon CrawlBuilds Common Crawl's open web archive, a dataset widely used to train language models.`CCBot/2.0 (https://commoncrawl.org/faq/)`	Model training	Your pages stay out of future Common Crawl snapshots, and so out of the many models trained on them.	Yes, including Crawl-delay; backs off on 429 and 5xx	Reverse DNS to crawl.commoncrawl.org, or the IP list · Docs	2.2%
DuckAssistBotDuckDuckGoCrawls in real time for DuckDuckGo's AI-assisted answers, which cite sources. DuckDuckGo says the data is not used to train AI models.`DuckAssistBot/1.2; (+http://duckduckgo.com/duckassistbot.html)`	AI search index	You lose citations in DuckDuckGo's AI answers. DuckDuckGo says organic rankings are unaffected.	Yes (changes take 72 hours)	Published IP list · Docs	0.5%
MistralAI-UserMistralVisits a page when a user of Mistral's assistant asks a question, and links to the source. Mistral says it is not used for training.`Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-User/1.0; +https://docs.mistral.ai/robots)`	User-triggered fetch	Mistral's assistant can't read or cite your page live.	Mistral documents it as robots.txt-controllable	Published IP list · Docs	0.2%
MistralAI-IndexMistralCrawls automatically to build Mistral's search index. Mistral says this content is not used for training.`Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-Index/1.0; +https://docs.mistral.ai/robots)`	AI search index	You drop out of Mistral's search index.	Yes	Published IP list · Docs	–
MistralAI-TrainingMistralCrawls to build datasets for training Mistral's models.`Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-Training/1.0; +https://docs.mistral.ai/robots)`	Model training	Keeps your future pages out of Mistral training data.	Yes	No IP list published · Docs	–
cohere-aiCohereA user agent seen on many sites (including ours) and attributed to Cohere. We could not find official Cohere documentation for it.`Contains “cohere-ai” (unverified)`	Model training	Unknown. Low volume in our data.	Not documented by Cohere	No published method · No official docs	0.3%

Loading tool…

Three kinds of AI crawler

Kind	What it does	Examples	Block it and you…
AI search index	Crawls ahead of time so an assistant can cite and link you	OAI-SearchBot, Claude-SearchBot, PerplexityBot, DuckAssistBot, MistralAI-Index	drop out of that assistant's citations
User-triggered fetch	Loads one page because a user asked something, right now	ChatGPT-User, Claude-User, Perplexity-User, MistralAI-User	stop the assistant reading you for that user, if it obeys the rule at all
Model training	Collects pages that may train future models	GPTBot, ClaudeBot, CCBot, meta-externalagent, Bytespider, MistralAI-Training	opt out of training, with no documented effect on search
robots.txt token only	Not a crawler: a rule that governs how another crawler's data is used	Google-Extended, Applebot-Extended	opt out of that company's AI training and grounding while staying in its search

Two big operators don't fit neatly. Google runs its AI Overviews and AI Mode on the regular Search index, so Googlebot is effectively its AI search crawler, and blocking it removes you from Google entirely. Google-Extended is the only lever for Gemini training. Microsoft grounds Copilot in Bing's index and has no separate Copilot token, so Bingbot does both jobs.

Why the user agent isn't enough

A user agent is a line of text any script can send. In our Observatory data, 416 requests across 10 of 12 sites claimed to be "Google-Extended", which Google says never appears as a user agent. If you're making decisions from your logs (or blocking at a firewall), verify by IP:

Published IP lists: OpenAI (one per agent, e.g. gptbot.json), Anthropic (bots.json), Perplexity (perplexitybot.json), DuckDuckGo, Mistral, Amazon and Common Crawl.
Reverse DNS: Google (hosts ending in googlebot.com), Apple (applebot.apple.com), Bing (search.msn.com) and Common Crawl (crawl.commoncrawl.org). Look up the hostname for the IP, then confirm that hostname resolves back to the same IP.
Nothing published: Meta documents its user agents but no IP method. ByteDance documents nothing for Bytespider.

Seen in our data but not listed

A few agents fetched pages on five or more of our sites but don't have operator documentation we could verify, so they're in the Observatory but not in the table: YouBot (You.com), DeepSeekBot and VelenPublicWebCrawler. We'll add them when we can source their documentation. We also left out agents that fetch only link previews (facebookexternalhit, Slackbot and the like), which aren't AI crawlers.

How we keep this current

We recheck every operator's documentation each quarter, alongside each Observatory edition, and move the "Updated" date at the top of the page when we do. Operators say version numbers in their user agents can change (OpenAI's docs list GPTBot/1.4 as of October 2026), so match on the product token, such as GPTBot or ClaudeBot, rather than the full string.

Questions people ask

What is the difference between GPTBot, OAI-SearchBot and ChatGPT-User?

GPTBot collects pages that may train OpenAI's models. OAI-SearchBot builds the index ChatGPT search cites. ChatGPT-User fetches a page live when a user asks about it. OpenAI says robots.txt may not apply to ChatGPT-User because a person started the request.

Is Google-Extended a crawler?

No. Google says Google-Extended has no separate user agent. It's a robots.txt token that controls whether content Googlebot already crawls may be used for Gemini training and grounding, and it doesn't affect Google Search.

Which AI crawler should I block to stop AI training but stay in AI search?

Block the training agents (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, meta-externalagent, Bytespider) and leave the search agents (OAI-SearchBot, Claude-SearchBot, PerplexityBot, DuckAssistBot, Googlebot, Bingbot) allowed.

How do I know a request is really from OpenAI or Anthropic?

Check the source IP against the operator's published list (for example openai.com/gptbot.json or claude.com/crawling/bots.json), or use reverse DNS where the operator supports it, as Google, Apple, Bing and Common Crawl do. User agents alone can be faked.