AI Search Observatory · Q4 2026 edition
What AI actually does on small‑business sites
Small businesses now get found three ways: Google, AI answers and their own website. Everyone has advice about the middle one. We publish measured data from real sites, and tools to check your own.
The gap
120:1
For every visit an AI assistant sent to these sites, AI crawlers fetched 120 pages. Search engines sent 33 times as many visitors as AI assistants did. AI is clearly reading these sites, and the part you can still influence is whether it names you when someone asks.
How we measured thisFree tool
Can AI read your site?
In about five seconds: which AI crawlers your robots.txt lets in, whether your homepage has real text without JavaScript, and whether anything tells AI to stay quiet. Every result comes with a plain-English fix.
Who's reading
Six agents did most of the fetching
Mean share of each site's AI fetches. A third of AI traffic was user-triggered: an assistant fetching a page live because someone had just asked about it.
- Training crawl
- Search index
- User-triggered fetch
Three ways a small business gets found
That's the three in Three Impact. Each needs something different from your site.
- 01
Google
Still where most found-you visits come from. AI Overviews and AI Mode are built from the same index, so Googlebot is an AI crawler too.
How AI picks businesses - 02
AI answers
ChatGPT, Claude, Perplexity, Gemini and Copilot read your site constantly and send back very few visitors. Being named in the answer is the win.
Every AI crawler, verified - 03
Your website
The one surface you control. If the text isn't in the HTML, or robots.txt says no, the other two never see you.
Platforms by AI-readiness
AI visibility tools, scored
One rubric, sourced prices, checked Nov 1, 2026. Weighted for a small business buying its first tool. We own cite-met, so we don't score it.
- 4.1Otterly.AIA single business or consultant tracking its own name and a few competitors.$29/mo→
- 3.9Peec AIAgencies and in-house teams tracking a few brands or markets, who want daily data and a tidy UI.$95/mo→
- 3.8ScrunchMulti-location and mid-size brands that want prompt tracking and agent analytics in one tool.$300/mo→
- 3.3ProfoundLarge brands and enterprise marketing teams with budget and someone to act on the data.Custom→
- 2.9Ahrefs Brand RadarSEO teams and agencies on Ahrefs doing competitive and market research.$199/mo→
- 2.8Semrush AI Visibility ToolkitTeams already on Semrush who want AI visibility in the same reports.$99/mo→
- Ourscite-metSmall businesses rebuilding or moving their site who want hosting and AI crawler data in one place.$60/mo→
Everything we publish
Updated quarterly. Every price and user agent links to its source.
- 01Observatory reportThe AI Search Observatory: what AI crawlers did on 12 small-business sitesMeasured, not modelled: AI crawlers fetched small-business sites 4.4x as often as Googlebot, yet sent 1 visit per 120 fetches. Q4 2026 data and method.
- 02Crawler registryAI crawler registry: every AI bot, its user agent and what blocking it costsGPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and 21 more: user agents, robots.txt tokens, verification links and observed share.
- 03Free toolAI access checker: can ChatGPT, Claude and Perplexity read your site?Free check of your robots.txt against 21 AI crawlers and tokens, plus noindex, nosnippet, text-without-JavaScript, schema and llms.txt, with a fix for each.
- 04Free toolllms.txt generator and validatorBuild a correct llms.txt in two minutes or check the one you have, against the llmstxt.org format. Runs in your browser; nothing is sent or stored.
- 05Guiderobots.txt for AI crawlers: what blocking each one costs youWhich AI bots to block and which to keep: the cost of blocking training, search and live-fetch crawlers, with copy-paste robots.txt recipes for small business.
- 06Evidence reviewDoes llms.txt matter? What the evidence says in 2026Google says it doesn't use llms.txt and no major assistant documents reading it. Who supports it, who reads it, and whether a small business should bother.
- 07Evidence reviewHow AI assistants pick which businesses to recommendWhat the evidence says about how ChatGPT, Google AI Overviews, Copilot and Perplexity pick businesses: Ahrefs studies, Google's guidance, BrightLocal, our data.
- 08ExplainerAI Overviews vs AI Mode: what changes for a small businessHow Google's AI Overviews and AI Mode differ, what the click and local-results studies show, and what a small business should do about each (mostly the same).
- 09GuideHow to track AI visibility: do it yourself, or pay for a tool?A free monthly method using Search Console, Bing, GA4 and a prompt sheet, where it breaks down, and when a paid AI visibility tracker earns its cost.
- 10How-toHow to track ChatGPT traffic in GA4, step by stepFind ChatGPT, Gemini, Claude and Perplexity visits in GA4: the AI Assistant channel, a reusable exploration, a regex, and why the numbers undercount.
- 11ReviewsAI visibility tool reviews: eight tools, one rubric, every price sourcedEight AI visibility tools scored on one published rubric, with sourced prices (Nov 2026), a sortable table, what 20 prompts really cost and the pick by need.
- 12CompareCompare AI search tools and platforms, head to headHead-to-head comparisons of AI visibility tools (Profound, Peec AI, Otterly, Scrunch, Semrush, Ahrefs), each with a verdict up front and sourced prices.
- 13How AI picks sourcesHow ChatGPT, Perplexity, Google AI, Copilot and Claude choose their sourcesWhere each AI assistant gets web results, which crawler to allow, how citations work and how often they match Google, from official docs and large studies.
- 14ComparisonWebsite platforms ranked by AI-readiness: Wix, Squarespace, Webflow, Framer, WordPress, Shopify and moreWhich website platforms serve real HTML to AI crawlers and let you control robots.txt, llms.txt and schema: Wix, Squarespace, Webflow, WordPress, Shopify.