The AI Search Observatory: what AI crawlers did on 12 small-business sites

This is the first edition of the Observatory. It is small on purpose: 12 sites, measured from the server side, with every figure aggregated across at least five of them. We'd rather publish a narrow sample we can explain than a big one we can't. The method and its limits are at the bottom of the page, and they are worth reading before you quote anything.

Three Impact is published by the team behind cite-met, the hosting and analytics platform these sites run on. That's how we can see the requests at all. It also means you should read this as data from one platform's customers, not from the web at large.

AI crawlers out-fetched Googlebot on every site

Between 1 August and 30 October 2026, AI crawlers made 20,806 requests to the 12 sites. Googlebot made 4,662. The median site saw AI crawlers 4.4 times as often as Googlebot, and there was no site in the sample where Google came out ahead.

1. All AI crawlers20,806
2. Amazonbot7,102
3. Googlebot4,662
4. Bingbot4,029
5. Applebot2,287

Show the numbers

Crawler	Fetches	Notes
All AI crawlers	20,806	every agent in the bot chart
Amazonbot	7,102	Amazon; mixed use, see registry
Googlebot	4,662	Google Search (and AI Overviews)
Bingbot	4,029	Bing (and Copilot)
Applebot	2,287	Apple (Siri, Spotlight)

Source: Three Impact AI Search Observatory, Q4 2026 edition. 12 small-business sites on cite-met. Googlebot means the main Googlebot user agent only.

Amazonbot sits in its own row because Amazon describes it as mixed use (it improves Amazon products and "may be used to train Amazon AI models"), so we didn't fold it into the AI total. Add it in and the gap gets wider.

For scale, Vercel's 2024 study of its own network found GPTBot, Claude, AppleBot and PerplexityBot together at about 28% of Googlebot's volume (Vercel, Dec 2024). The two numbers measure different bots, different years and very different sites, so they aren't directly comparable. It's still a useful reminder that small sites with a few dozen pages get re-read by AI agents far more often than their size would suggest.

Why they came: training, search, or someone asking

Each AI agent documents a purpose, so we sorted every fetch into one of three buckets using the operator's own description (see the crawler registry for the sources):

• Training crawls collect pages that may be used to train models. GPTBot, ClaudeBot, CCBot, Meta-ExternalAgent and Bytespider are the big ones.
• Search indexing builds the index an assistant cites from, such as OAI-SearchBot, PerplexityBot and Claude-SearchBot.
• User-triggered fetches happen when a person asks an assistant something and it loads your page right then (ChatGPT-User, Claude-User, Perplexity-User).

40.4%24.5%35.1%

• Training crawl
• Search index
• User-triggered fetch

Show the numbers

Purpose	Share of AI fetches
Training crawl	40.4%
Search index	24.5%
User-triggered fetch	35.1%

Source: Three Impact AI Search Observatory, Q4 2026 edition. 12 small-business sites on cite-met; site-weighted. Purpose follows each operator's own documentation for the user agent.

The surprise for us was the size of the third bucket. On the typical site, 35% of AI fetches were made on behalf of a person who had just asked a question. Those fetches are the closest thing to a customer reading your site through an assistant, and several operators say robots.txt may not apply to them because a human started the request. Blocking training crawlers is a reasonable choice. Making sure the live-fetch agents find real text when they arrive matters more.

Which agents did the fetching

1. ChatGPT-UserOpenAI28.1%
2. OAI-SearchBotOpenAI14.9%
3. ClaudeBotAnthropic14.7%
4. GPTBotOpenAI10.1%
5. Meta-ExternalAgentMeta7.2%
6. PerplexityBotPerplexity6.9%
7. BytespiderByteDance4.5%
8. Claude-UserAnthropic4.2%
9. CCBotCommon Crawl2.2%
10. Perplexity-UserPerplexity2.1%
11. Claude-SearchBotAnthropic1.4%
12. DeepSeekBotDeepSeek0.8%
13. YouBotYou.com0.7%
14. DuckAssistBotDuckDuckGo0.5%
15. VelenPublicWebCrawlerVelen0.5%
16. Google-CloudVertexBotGoogle0.3%
17. cohere-aiCohere0.3%
18. MistralAI-UserMistral0.2%

• Training crawl
• Search index
• User-triggered fetch

Show the numbers

User agent	Operator	Purpose	Sites (of 12)	Share
ChatGPT-User	OpenAI	User-triggered fetch	12	28.1%
OAI-SearchBot	OpenAI	Search index	12	14.9%
ClaudeBot	Anthropic	Training crawl	12	14.7%
GPTBot	OpenAI	Training crawl	12	10.1%
Meta-ExternalAgent	Meta	Training crawl	11	7.2%
PerplexityBot	Perplexity	Search index	12	6.9%
Bytespider	ByteDance	Training crawl	10	4.5%
Claude-User	Anthropic	User-triggered fetch	12	4.2%
CCBot	Common Crawl	Training crawl	10	2.2%
Perplexity-User	Perplexity	User-triggered fetch	8	2.1%
Claude-SearchBot	Anthropic	Search index	8	1.4%
DeepSeekBot	DeepSeek	Training crawl	8	0.8%
YouBot	You.com	Search index	9	0.7%
DuckAssistBot	DuckDuckGo	Search index	9	0.5%
VelenPublicWebCrawler	Velen	Training crawl	7	0.5%
Google-CloudVertexBot	Google	User-triggered fetch	7	0.3%
cohere-ai	Cohere	Training crawl	6	0.3%
MistralAI-User	Mistral	User-triggered fetch	5	0.2%
Others (on fewer than 5 sites)	various	various	<5 each	0.5%

Source: Three Impact AI Search Observatory, Q4 2026 edition. 12 small-business sites on cite-met; site-weighted. Agents seen on fewer than 5 sites are merged into “others” (0.5%).

OpenAI dominates. ChatGPT-User alone was 28.1% of the typical site's AI fetches, and OpenAI's three agents together came to 53%. Anthropic was second at 20.4%, mostly ClaudeBot training crawls. Perplexity was third. Google barely registers here, because its AI features run on ordinary Googlebot, which we count separately above. Gemini and AI Overviews don't need a separate crawler to read you.

1. OpenAI53.0%
2. Anthropic20.4%
3. Perplexity8.9%
4. Meta7.4%
5. ByteDance4.6%
6. Common Crawl2.2%
7. All others3.5%

Show the numbers

Operator	Share	Sites (of 12)
OpenAI	53.0%	12
Anthropic	20.4%	12
Perplexity	8.9%	12
Meta	7.4%	11
ByteDance	4.6%	10
Common Crawl	2.2%	10
All others	3.5%	12

Source: Three Impact AI Search Observatory, Q4 2026 edition. 12 small-business sites on cite-met; site-weighted.

Agents seen on fewer than five sites are merged into "others" so that no bucket can describe a single site. That rule hides some long-tail bots, including Meta's live-fetch agent, which showed up rarely.

The gap: AI reads a lot and sends almost no one

Reading isn't the same as recommending, and recommending isn't the same as a visit. From 9 August, when cite-met started separating AI crawler requests from visitors who arrived from an AI assistant, to 30 October, the 12 sites logged:

1. AI crawler fetches19,785
2. Visits from search engines5,473
3. Visits from AI assistants165

Show the numbers

Measure	Total (12 sites)	Median per site
AI crawler fetches	19,785	–
Visits from search engines	5,473	305
Visits from AI assistants	165	7.5

Source: Three Impact AI Search Observatory, Q4 2026 edition. 12 small-business sites on cite-met. A referral is a visit whose referrer was an AI assistant or search engine.

That's 120 AI crawler fetches for every visit an AI assistant sent, and search engines sent 33 times as many visitors as AI assistants did. The median site received about seven AI-referred visits in 83 days, which works out to roughly one every 11 days, and AI assistants accounted for 0.03% of its visits.

We think two things are going on, and we can't yet tell you how much each contributes. Assistants answer a lot of questions without sending anyone anywhere, so a business can be read, and even named, without a click. Assistant apps also often hide the referrer, so some AI-driven visits will show up as direct traffic and we undercount them. Either way, if you are judging AI search by referral visits in your analytics, you'll see close to nothing, and that number says little about whether assistants are naming you.

AI crawling tripled by August, then eased

To see a trend without new sites joining and inflating the line, we followed a fixed cohort: the eight sites that had data in every month from May to October.

2,174

549May

4,028

1,244Jun

5,032

1,414Jul

6,422

1,230Aug

4,344

862Sep

4,963

1,105Oct

• Training crawl
• Search index
• User-triggered fetch

Googlebot fetches, same sites, for reference

Show the numbers

Month	Training	Search	User-triggered	All AI	Median per site	Googlebot
May	1,254	205	715	2,174	97	549
Jun	1,577	546	1,905	4,028	269	1,244
Jul	2,176	848	2,008	5,032	444	1,414
Aug	2,585	1,203	2,634	6,422	698	1,230
Sep	1,769	1,011	1,564	4,344	474	862
Oct	2,629	947	1,387	4,963	342	1,105

Source: Three Impact AI Search Observatory, Q4 2026 edition. The 8 sites with data in every month; October runs to the 30th.

AI fetches roughly tripled from May to August, from 2,174 to 6,422 a month across the cohort, then dropped back to about 4,300 to 5,000 a month in September and October. Search-index crawling grew fastest in relative terms, about six times from May to August. The median site went from 97 AI fetches in May to 698 in August.

Treat the May number with some care, since one cohort site only had a few days of data that month. The shape (a steep spring climb, then a plateau) holds with or without it.

Fake "Google-Extended" requests

Google says Google-Extended "doesn't have a separate HTTP request user agent string" (Google). It's a robots.txt token, nothing more. Even so, 416 requests to 10 of our 12 sites arrived claiming to be Google-Extended in their user agent. None of them can have come from Google. We left them out of every figure above.

It's a small number, but it's the cleanest evidence in our data that user agents get spoofed, and it's why the registry lists each operator's IP verification method next to every bot.

What this means for a small business

• Leave AI search crawlers alone. OAI-SearchBot, Claude-SearchBot, PerplexityBot and DuckAssistBot are how assistants find you to cite you. Block training bots if you like; our robots.txt guide has copy-paste versions.
• Put the facts in the HTML. A third of AI fetches are an assistant reading your page live for a customer. Vercel found the major AI crawlers don't run JavaScript, so services, area served, prices and hours need to be in the page as delivered. The access checker shows what they see.
• Don't judge AI by referral visits. With a median of 0.03% of visits, referral counts can't tell you whether assistants recommend you. Ask them directly, or use one of the AI visibility trackers we compare.
• Keep Google first. Search engines still sent 33 times as many visitors, and Google's AI features draw on the same index.

Methodology

Source. cite-met's production analytics database, which rolls up server-side request logs into daily counts per site, per bot and per traffic channel. We ran read-only queries on 31 October 2026 and froze the results into this page. Nothing here is sampled or modelled.

Sites. 12 small-business websites hosted on cite-met with a verified custom domain. We excluded sites run by the team behind Three Impact (agency, product and personal sites), demo and preview sites without a custom domain, non-business app subdomains, sites with fewer than 30 days of data, and externally hosted sites that cite-met only monitors through a JavaScript beacon, which can't see crawlers. We also excluded one very large site that on its own produced most of the network's traffic and about nine in ten of its AI referrals. Including it would have turned this report into a report about one site.

Windows. Crawler figures cover 1 August to 30 October 2026 (91 days). Referral figures start on 9 August 2026, when cite-met began splitting AI crawler requests from AI-referred visits. The trend uses May to October for the fixed eight-site cohort.

Classification. Requests are classified by user agent against a maintained registry of known bots. A request counts as an AI fetch when its agent is one of the documented AI crawlers or fetchers in our registry. A visit counts as AI-referred when it's a non-bot request whose referrer is a known AI assistant; search-referred is the same for search engines.

Weighting. Shares (by purpose, bot and operator) are site-weighted: we compute each site's share, then average across sites, so a busy site can't drive the result. Totals are pooled sums, and we say so wherever we use one.

Privacy. We publish only aggregates. Every figure on this page covers at least five sites; any bot or bucket seen on fewer is merged into "others" or dropped. No site, client, industry or location is named or implied, and no per-site number appears.

Limitations

• Twelve sites is a small sample, and every one is on the same platform. They are all small businesses, so treat the findings as indicative.
• User agents can be faked. We classify by user agent, not by verified IP. We dropped requests claiming to be Google-Extended or Applebot-Extended, which can't be genuine, but other spoofed requests may remain.
• AI referrals are undercounted. Many assistant apps strip the referrer, so some AI-driven visits are counted as direct traffic.
• We can't see mentions. This data shows what AI agents did on the sites, not what the assistants said about them afterwards.
• We don't break AI referral visits out by assistant in this edition. The rollups we used store AI referrals as one channel.
• We can't yet count llms.txt or robots.txt requests. The path-level rollups record page views, not requests for machine-readable files. We plan to add that for the next edition.
• "Human" visits include some unidentified bots. Bots that don't announce themselves look like people, which makes AI's share of visits look smaller than it is.

Using this data

You're welcome to quote and chart these figures with a link to this page. Please keep the sample size (12 sites) and the period next to any number you use. The next edition is planned for the first quarter of 2027, with a larger sample and the llms.txt and robots.txt request counts this one lacks.