AI Search Observatory · Q4 2026 edition
What AI actually does on small‑business sites
Small businesses now get found three ways: Google, AI answers and their own website. Everyone has advice about the middle one. We publish measured data from real sites, and tools to check your own.
The gap
120:1
For every visit an AI assistant sent to these sites, AI crawlers fetched 120 pages. Search engines sent 33 times as many visitors as AI assistants did. AI is clearly reading these sites, and the part you can still influence is whether it names you when someone asks.
How we measured thisFree tool
Can AI read your site?
In about five seconds: which AI crawlers your robots.txt lets in, whether your homepage has real text without JavaScript, and whether anything tells AI to stay quiet. Every result comes with a plain-English fix.
Who's reading
Six agents did most of the fetching
Mean share of each site's AI fetches. A third of AI traffic was user-triggered: an assistant fetching a page live because someone had just asked about it.
- Training crawl
- Search index
- User-triggered fetch
Three ways a small business gets found
That's the three in Three Impact. Each needs something different from your site.
- 01
Google
Still where most found-you visits come from. AI Overviews and AI Mode are built from the same index, so Googlebot is an AI crawler too.
How AI picks businesses - 02
AI answers
ChatGPT, Claude, Perplexity, Gemini and Copilot read your site constantly and send back very few visitors. Being named in the answer is the win.
Every AI crawler, verified - 03
Your website
The one surface you control. If the text isn't in the HTML, or robots.txt says no, the other two never see you.
Platforms by AI-readiness
Everything we publish
Updated quarterly. Every price and user agent links to its source.
- 01Observatory reportThe AI Search Observatory: what AI crawlers did on 12 small-business sitesMeasured, not modelled: AI crawlers fetched small-business sites 4.4x as often as Googlebot, yet sent 1 visit per 120 fetches. Q4 2026 data and method.
- 02Crawler registryAI crawler registry: every AI bot, its user agent and what blocking it costsGPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and 21 more: user agents, robots.txt tokens, verification links and observed share.
- 03Free toolAI access checker: can ChatGPT, Claude and Perplexity read your site?Free check of your robots.txt against 21 AI crawlers and tokens, plus noindex, nosnippet, text-without-JavaScript, schema and llms.txt, with a fix for each.
- 04Guiderobots.txt for AI crawlers: what blocking each one costs youWhich AI bots to block and which to keep: the cost of blocking training, search and live-fetch crawlers, with copy-paste robots.txt recipes for small business.
- 05Evidence reviewDoes llms.txt matter? What the evidence says in 2026Google says it doesn't use llms.txt and no major assistant documents reading it. Who supports it, who reads it, and whether a small business should bother.
- 06Evidence reviewHow AI assistants pick which businesses to recommendWhat the evidence says about how ChatGPT, Google AI Overviews, Copilot and Perplexity choose businesses: Ahrefs' studies, Google's guidance, BrightLocal and our data.
- 07ComparisonAI visibility (AEO) tracking tools compared: prices, what they measure, who they're forProfound, Peec AI, Otterly.AI, Semrush AI Visibility Toolkit, Ahrefs Brand Radar, Scrunch and cite-met: sourced prices (Oct 2026), what each measures, who it fits.
- 08ComparisonWebsite platforms ranked by AI-readiness: Wix, Squarespace, Webflow, Framer, WordPress, Shopify and moreWhich website platforms serve real HTML to AI crawlers and let you control robots.txt, llms.txt and schema. Wix, Squarespace, Webflow, Framer, WordPress, Shopify, SPAs.