For a typical small-business site, no. The median site in our Observatory got about 513 AI crawler fetches a month, which at a normal HTML page size is roughly 11-18 MB a month: about what four to seven visitors use loading a typical homepage with its images. The bandwidth horror stories are real, but they come from huge sites with millions of pages or large downloads. If your host bills by bandwidth and a crawler misbehaves, slow it down before you block it, because the same bots decide whether AI assistants can cite you.
What we measured
Across the 12 small-business sites in the Q4 2026 Observatory, AI crawlers made 20,806 fetches between 1 August and 30 October 2026. The median site received about 513 a month, 4.4 times its Googlebot volume. These come from server-side logs, so they count crawlers that never run JavaScript and never show up in Google Analytics. The full method and limits are on the Observatory page, and the per-bot table is a free CSV download.
How we got to 11-18 MB
We don't log bytes per request, so the bandwidth figure is an estimate, and here is how it's built. The HTTP Archive Web Almanac 2025 puts the median home page at 2.86 MB on desktop and 2.56 MB on mobile, but almost all of that is images, scripts and fonts. The HTML itself is far smaller: a median of 22 KB of HTML in the Almanac's content-type breakdown, and 33-35 KB in its HTML-size distribution. Crawlers that read text fetch the HTML, so 513 fetches at 22-35 KB is about 11-18 MB a month.
Two things push the real number up: crawlers that also fetch images or PDFs, and very heavy HTML (page builders can produce 200 KB pages or more). Even at 200 KB a page, 513 fetches is about 100 MB a month, which most hosting plans won't notice.
When AI crawlers do cost real money
The cases on record are large sites:
- Read the Docs reported in July 2024 that one crawler downloaded 73 TB of zipped HTML in May 2024, which cost over $5,000 in bandwidth charges. Blocking it and similar crawlers cut its download bandwidth by 75% (Read the Docs).
- Wikimedia said in April 2025 that bandwidth for multimedia downloads had grown 50% since January 2024, and that bots made up at least 65% of its most resource-consuming traffic while being about 35% of pageviews (Wikimedia Foundation).
- SourceHut's founder wrote in March 2025 that aggressive LLM crawlers took 20-100% of his time in a given week and caused dozens of brief outages a week (Drew DeVault). That is an outage and staff-time problem rather than a bill.
What these sites share: millions of pages or big downloadable files. A 30-page business site is a different situation.
How to spot AI crawlers in your logs
Look for the product token in the user agent, then confirm the IP. User agents can be faked, and the SourceHut post describes crawlers arriving from tens of thousands of residential IPs, so a published IP list only confirms the honest ones.
| Operator | Tokens to look for | How to verify |
|---|---|---|
| OpenAI | GPTBot, OAI-SearchBot, ChatGPT-User |
IP lists: gptbot.json, searchbot.json, chatgpt-user.json |
| Anthropic | ClaudeBot, Claude-SearchBot, Claude-User |
IP list: claude.com/crawling/bots.json |
| Perplexity | PerplexityBot, Perplexity-User |
IP lists: perplexitybot.json, perplexity-user.json |
Googlebot (and Google-Extended is not a crawler) |
Reverse DNS to googlebot.com, google.com or googleusercontent.com, then forward DNS (Google) | |
| Microsoft | bingbot |
Reverse DNS ending in search.msn.com, or the bingbot.json list |
Our AI crawler registry has 32 AI agents and robots.txt tokens with their documented user agents and verification links, and the same list as JSON or CSV if you want to load it into a log tool. One thing we found in our own logs: requests claiming to be "Google-Extended" appeared on 10 of 12 sites, although Google says Google-Extended has no user agent of its own. Those requests are not from Google.
Slow them down before you block them
Blocking the wrong crawler costs you AI citations (see what blocking each crawler costs). If bandwidth is the worry, try these first:
- Crawl-delay for Anthropic. Anthropic says its crawlers support the non-standard
Crawl-delayrule. Google says it does not support crawl-delay, and OpenAI and Perplexity don't document it, so it only helps for some bots. - Rate limits at your host or CDN. On Cloudflare, rate limiting and AI Crawl Control's per-crawler Bytes figures let you see and cap one crawler without blocking it. Our Cloudflare guide covers which settings block AI search by accident.
- Block training only. If you'd rather not feed model training at all, block GPTBot, ClaudeBot, CCBot and the other training agents and keep the search agents open. That removes about 40% of AI fetches in our sample, the training share, at no documented cost in AI search.
- Make pages lighter. Serving clean HTML helps your visitors and every crawler, and keeps the bytes per fetch near the 22-35 KB median.
Questions people ask
Do AI crawlers increase hosting bandwidth costs for a small business?
Rarely by much. In our Q4 2026 sample the median small-business site received about 513 AI crawler fetches a month. At a median HTML size of 22-35 KB a page, that is roughly 11-18 MB of HTML a month, an estimate that assumes crawlers fetch HTML rather than every image.
Can I slow AI crawlers down instead of blocking them?
Partly. Anthropic says its crawlers support the non-standard Crawl-delay rule in robots.txt. Google says it does not support crawl-delay, and OpenAI and Perplexity don't document it. For the others, rate limiting at your host or CDN is the dependable option.
How do I tell which requests in my logs are AI crawlers?
Match the user agent token (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot and so on), then confirm the IP against the operator's published list, such as openai.com/gptbot.json or claude.com/crawling/bots.json. Google and Bing are verified by reverse DNS. A user agent alone can be faked.
Which sites have had real bandwidth problems from AI crawlers?
Large content and code hosts. Read the Docs reported one crawler downloading 73 TB in May 2024 at a cost of over $5,000, and Wikimedia reported in April 2025 that bots were at least 65% of its most resource-consuming traffic.