What the checker looks at
It makes three requests to the domain you type, and nothing else: /robots.txt, the homepage, and /llms.txt. Each request has a five-second limit and reads at most 512 KB. It follows redirects only within the same site (for example from http to https, or to the www version), and it won't fetch IP addresses, private hosts or non-standard ports. The check runs on our server, so your browser never talks to the site.
| Check | Why it matters | What "good" looks like |
|---|---|---|
| robots.txt against 21 agents | Decides which AI crawlers are asked to stay away | Search agents allowed; training agents your choice |
Blanket User-agent: * block |
Blocks every bot without its own group, including search | No Disallow: / under * |
| X-Robots-Tag and meta robots | noindex removes you from Google, including AI Overviews |
No noindex or none on the homepage |
nosnippet |
Keeps your text out of snippets and AI Overviews, and out of Apple's AI answers | Use max-snippet or data-nosnippet for specific passages instead |
| Text without JavaScript | Most AI crawlers don't run scripts, so a script-only page looks empty to them | Hundreds of words of real text in the delivered HTML |
| JSON-LD structured data | Not needed for AI features, but states your facts unambiguously | A LocalBusiness or Organization block |
| llms.txt | A proposed file no major assistant documents using | Optional either way |
How to read each result
AI search crawlers blocked. This is the one to fix first. OAI-SearchBot, Claude-SearchBot, PerplexityBot and DuckAssistBot build the indexes their assistants cite from, so a block here means those assistants can't link to you from their index. It usually comes from a "block AI bots" setting or plugin that lumps search and training together. Remove the Disallow for those tokens. Our robots.txt guide has a version that blocks training and keeps search.
Training opt-outs in place. GPTBot, ClaudeBot, CCBot, Google-Extended and the others in the training group only affect whether your future pages train models. OpenAI says GPTBot is separate from ChatGPT search, Google says Google-Extended doesn't affect Search, and Apple says the same of Applebot-Extended. It's your call, and we show it as information, not a problem.
Live-fetch agents blocked. ChatGPT-User, Claude-User and Perplexity-User fetch a page when someone asks an assistant about it. OpenAI and Perplexity both say robots.txt may not apply to these because a person started the request, so the block may do nothing. Where it does work, it stops assistants reading your page for a customer who asked.
Only a few words without JavaScript. If the homepage arrives as an empty shell that a script fills in, crawlers that don't run JavaScript see nothing to quote. Vercel's study of AI crawler traffic found that none of the major AI crawlers it measured render JavaScript (Vercel). Google's crawler does, which is why a site can rank in Google and still be invisible to ChatGPT. The fix is server-side rendering or pre-rendering; our platform comparison shows which builders do it for you.
noindex on the homepage. Google requires a page to be indexed and eligible for a snippet before it can appear in AI Overviews or AI Mode (Google). A stray noindex is common after a site launch, when a "discourage search engines" setting gets left on.
No llms.txt. That's fine. Google says you don't need AI text files to appear in its AI features, and we cover the rest of the evidence in Does llms.txt matter?
What it can't see
- Firewall and CDN blocks. Some hosts and CDNs can block AI crawlers by user agent or IP before robots.txt is ever read. We fetch as our own user agent, so a site can pass here and still turn GPTBot away at the network edge. Your host's bot settings or firewall logs will tell you.
- Pages other than the homepage. Robots rules are tested for
/. A rule that blocks/services/shows as "allowed, some paths blocked". - What assistants say about you. Access is the first step, not the result. To see whether ChatGPT or Perplexity name you, ask them, or use one of the AI visibility trackers we compare.
How we built it
The robots.txt matching follows RFC 9309: a bot uses the group that names its token, or the * group if none does, and the longest matching rule wins, with Allow winning a tie. A robots.txt that returns a server error is treated as "disallow everything", as the standard says; a 404 means no restrictions. The word count strips scripts, styles and markup from the delivered HTML and counts what's left. The list of agents comes from our crawler registry, where each one links to its operator's documentation.
Questions people ask
How do I check if ChatGPT can access my website?
Enter your domain in the checker above. It reads your robots.txt and reports whether OAI-SearchBot (ChatGPT search), GPTBot (training) and ChatGPT-User (live fetches) are allowed, and whether your homepage has readable text without JavaScript.
Does the checker store my website or results?
No. The check runs on request and nothing is saved. Results are cached for five minutes so repeat checks are fast.
Why does my site pass but AI still can't see it?
Robots.txt is only a request. A firewall or CDN setting that blocks AI bots by user agent or IP will stop them even when robots.txt allows them, and this checker fetches as itself, so it can't see those blocks.