AI Crawler Access Checker
Find out whether ChatGPT, Claude, Perplexity, Copilot and Gemini can actually read your site. We parse robots.txt the way crawlers do, show an allow/block matrix for every AI bot, and hand you a corrected policy.
Robots rules, per crawler
Longest-match wins and named groups override the wildcard — exactly as RFC 9309 reads it.
Answer engines first
ChatGPT search, Perplexity and Claude retrieval are scored apart from training crawlers.
A policy you can paste
Get a corrected snippet or a preset that allows assistants and blocks training.
Which AI crawler decides what
The single most common mistake is blocking GPTBot to “opt out of ChatGPT”. GPTBot only collects training data — the crawler that decides whether ChatGPT can cite you is OAI-SearchBot.
| Crawler | Owner | Purpose | Drives citations | If you block it |
|---|---|---|---|---|
| GPTBot | OpenAI | Model training | No | Blocking it is a training opt-out. ChatGPT can still read and cite the page through OAI-SearchBot. |
| OAI-SearchBot | OpenAI | ChatGPT search index | Yes | Blocking it removes the site from ChatGPT search results and citations. |
| ChatGPT-User | OpenAI | Live page fetch for a user | Yes | Blocking it stops ChatGPT from opening a URL a person pasted or asked about. |
| PerplexityBot | Perplexity | Perplexity index | Yes | Blocking it keeps every Perplexity answer from citing the page. |
| ClaudeBot | Anthropic | Model training | No | Training access only. Claude citations use Claude-SearchBot and Claude-User. |
| Claude-SearchBot | Anthropic | Claude search index | Yes | Blocking it hides the site from cited Claude answers. |
| Google-Extended | Gemini training and grounding | No | Does not remove the page from Google Search or AI Overviews — Googlebot does that. | |
| CCBot | Common Crawl | Open crawl dataset | No | Many models train on Common Crawl. Blocking it lowers training exposure, not citations. |
| Bytespider | ByteDance | Model training | No | Training crawler; unrelated to ChatGPT, Claude or Perplexity answers. |
How the score works
Answer-engine access (35 points). Each citation crawler carries a weight. Googlebot and the chat search crawlers matter most, because Google AI Overviews read the normal index and ChatGPT search reads OAI-SearchBot.
Indexability (30 points). A 200 status, no noindex in the meta tag or X-Robots-Tagheader, a self-referencing canonical, HTTPS, and an HTML content type.
Content clarity (20 points). Text in the served HTML, a descriptive title, a meta description, one H1, and JSON-LD.
AI guidance files (15 points). A valid /llms.txt, a reachable sitemap, and a robots.txt that answers.
Training-only crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, meta-externalagent) carry no score weight. Choosing to block them is legitimate; it just is not a citation problem.
The classic breakages
- 1A wildcard disallow. User-agent: * / Disallow: / blocks every AI crawler at once. Named groups override the wildcard, so the fix is to add explicit allow groups rather than editing the wildcard.
- 2Blocking GPTBot and expecting ChatGPT to stop citing you. It will not. Training and retrieval use different user agents. Block GPTBot, ClaudeBot and Google-Extended if you want, but leave OAI-SearchBot, PerplexityBot, Claude-SearchBot and Googlebot alone.
- 3A robots.txt that returns 500. RFC 9309 says a 5xx response should be treated as a temporary full block. A broken CDN rule or edge function can quietly remove the whole site from AI answers.
- 4Client-rendered content. Crawlers fetch HTML; most do not run your JavaScript. If the body text is empty in the raw response, the page effectively has no content to cite.
Where this fits
Crawler access is step one. Once AI assistants can read the site, check the AI technical readiness scorecard for schema, semantic signals and discovery files, then generate an llms.txt that points them at the right pages. Analytise then shows which AI assistants actually send visitors and convert.