Vero Beach, Florida · Organic & AI Search Optimization Services for Growth-Focused Businesses
Can AI find and trust your brand? Get your free Signal Score →
Home / AI Crawler Checker
Free tool · AI search visibility · Instant
AI Crawler Checker: Can ChatGPT, Claude, Perplexity and Google Actually Reach Your Site?
AI assistants can only cite pages their crawlers are allowed to read. This free checker tests 14 AI and search crawlers against your website in two ways: it reads your robots.txt the way the crawlers do, and it requests your homepage as each crawler to catch firewall and CDN rules that block them silently. Enter a domain below to see exactly who can reach you, which rule is responsible, and what to change.
- 14 crawlers from OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple and Common Crawl
- robots.txt evaluated to the official standard, rule by rule
- A live firewall test that robots.txt checkers miss
- Free, instant, no email required
Run the check
Check AI Crawler Access for Any Domain
Why it matters
If AI Crawlers Cannot Read You, AI Cannot Recommend You
When someone asks ChatGPT, Claude, Perplexity or Google’s AI Mode which company to hire, the answer is built from pages those systems were able to read. Each AI company uses its own crawlers, identified by a name such as OAI-SearchBot or PerplexityBot, and each crawler checks your robots.txt file before it fetches anything. A single line in that file, often added years ago or by a plugin, can quietly remove your business from AI answers.
The second, more common problem is invisible in robots.txt entirely. Content delivery networks and security services can block crawlers based on how they identify themselves. Cloudflare, for example, offers a one-click Block AI Bots setting. These rules are often switched on by a hosting provider, a developer or a security plugin, and the marketing team never knows. Your robots.txt can say “welcome” while your firewall says “403 Forbidden”.
That is why this checker runs both tests. It is the same check we run at the start of every AI search optimization engagement, because there is no point optimizing content that AI systems are not allowed to see.
What the Checker Reports
- How many AI search and answer crawlers can reach you
- Each crawler’s robots.txt verdict and the exact rule behind it
- Which crawlers your server refuses when they identify themselves
- Whether robots.txt is missing, broken or returning errors
- Whether you publish an llms.txt file
The crawlers
The 14 Crawlers We Test, and What Each One Does
Names and purposes come from each company’s own documentation. AI companies now separate their crawlers by job, so you can allow the ones that put you in answers and still opt out of model training.
| Crawler | Company | Job | If you block it |
|---|---|---|---|
| OAI-SearchBot | OpenAI | Finds and links sites in ChatGPT search | You stop appearing as a linked source in ChatGPT search answers |
| ChatGPT-User | OpenAI | Fetches a page when a ChatGPT user asks about it | ChatGPT cannot open your page when a user asks it to |
| GPTBot | OpenAI | Collects content to train OpenAI models | Your content is excluded from future OpenAI training |
| Claude-SearchBot | Anthropic | Indexes content to improve Claude’s search results | Lower visibility in Claude’s search results |
| Claude-User | Anthropic | Fetches a page when a Claude user asks about it | Claude cannot open your page for its users |
| ClaudeBot | Anthropic | Collects content for Anthropic model training | Your content is excluded from future Anthropic training |
| PerplexityBot | Perplexity | Finds and links sites in Perplexity answers (not used for training) | You stop appearing as a cited source in Perplexity |
| Perplexity-User | Perplexity | Fetches a page when a Perplexity user asks about it | Perplexity cannot open your page for its users |
| Googlebot | Google Search, including AI Overviews and AI Mode | You disappear from Google Search entirely | |
| Google-Extended | Control token for Gemini training and grounding in Gemini Apps | Excluded from Gemini training and grounding; Google Search is not affected | |
| Bingbot | Microsoft | Bing search, which also powers Microsoft Copilot answers | You disappear from Bing and lose Copilot visibility |
| Applebot | Apple | Siri, Spotlight and Safari suggestions | You drop out of Apple’s search features |
| Applebot-Extended | Apple | Control token for Apple foundation model training | Excluded from Apple model training; Siri and Spotlight are not affected |
| CCBot | Common Crawl | Builds an open web archive that many AI models train on | Your content is left out of future Common Crawl archives |
Google-Extended and Applebot-Extended are control tokens rather than separate crawlers. Google and Apple crawl with their normal bots and read these robots.txt entries to decide how the content may be used, so the firewall test does not apply to them.
Three kinds of crawler
Search, User and Training Crawlers Are Different Decisions
Treating every AI crawler the same is the most common mistake we see. The right setting depends on what each crawler is for.
Allow
Search Crawlers
OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot and Applebot decide whether you can be found and cited. For almost every business these should be open, because blocking them removes you from the answers your buyers read.
Allow
User-Initiated Fetchers
ChatGPT-User, Claude-User and Perplexity-User only visit when a person asks an assistant about a specific page. Blocking them means an assistant cannot read your pricing or services page even when a prospect asks it to.
Your choice
Training Crawlers
GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot gather content for model training. Blocking them is a legitimate choice for publishers protecting original work. It does not remove you from AI search, but Google-Extended also affects grounding in the Gemini app.
How the check works
We Read Robots.txt the Way the Crawlers Do
Many free robots.txt checkers simply look for a crawler’s name in the file. That gets the answer wrong surprisingly often. Our checker follows the Robots Exclusion Protocol standard (RFC 9309), which Google and the major AI companies follow. A crawler obeys the group of rules that names it and ignores the general “User-agent: *” group entirely. Rules from repeated groups for the same crawler are combined. When several rules match a path, the longest one wins, and an Allow rule wins a tie. Wildcards such as “*” and the end-of-path marker “$” are supported.
File errors matter too. If your robots.txt returns “not found”, crawlers treat the whole site as allowed. If it returns a server error or cannot be reached, crawlers treat the whole site as blocked, which is one of the fastest ways to vanish from search without anyone noticing.
The firewall test then requests your homepage twice: once as a normal browser and once identifying as each crawler that has its own user agent. If the browser gets the page but the crawler is refused with a 401, 403, 406, 451 or 503, or no response at all, something on your server or network is blocking that crawler. A 429 “too many requests” response is reported as inconclusive, because rate limits can be triggered by the test itself. Our requests do not come from each AI company’s published IP addresses, so treat a firewall warning as a strong lead to confirm in your CDN settings rather than final proof.
Fixing problems
How to Fix What the Checker Finds
Most fixes take minutes once you know where the block lives.
Fix Robots.txt Errors First
If your robots.txt returns a server error, every crawler treats your site as blocked. Make sure the file loads with a 200 status, or remove it entirely so it returns 404.
Remove Blocks on Search and User Crawlers
Delete any Disallow rules that apply to OAI-SearchBot, Claude-SearchBot, PerplexityBot, the user fetchers, Googlebot or Bingbot, including a site-wide “Disallow: /” in the “User-agent: *” group that they fall back to.
Check Your CDN and Security Settings
If the firewall test flags a crawler, look for AI bot blocking in your CDN, web application firewall or security plugin. Allow the crawlers you want, using each company’s published IP ranges where your provider supports verified bots.
Decide Deliberately on Training Crawlers
Choose whether to allow GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot as a business decision, write it into robots.txt, and re-run this check to confirm the result.
Beyond access
Access Is Step One. Being Chosen Is the Goal
Passing this check means AI systems can read your site. Whether they cite you depends on everything else: how much the wider web trusts your domain, whether your business is a clear and consistent entity, and whether your pages answer questions directly. That is what the SEOMA Signal Score measures, using real link-graph, Chrome user and crawl data, and what our answer engine optimization and AI Overviews optimization services improve.
On llms.txt: some AI tools read an llms.txt file that summarizes a site for language models. Google has said it does not use the file for Search, and it neither helps nor hurts Google rankings. It is cheap to publish and some other tools use it, so the checker reports whether you have one without scoring it.
Honest limits
What This Checker Cannot See
A useful tool is clear about where its evidence stops.
- Requests from the crawlers’ real IP addresses: some firewalls allow verified crawlers while refusing our test, so a firewall warning should be confirmed in your CDN settings
- Individual pages: the check covers your homepage and site-wide rules, not page-level noindex or nosnippet tags
- Whether AI assistants actually cite you today: that depends on authority and content, which the Signal Score and a GEO audit measure
- Crawlers that ignore robots.txt: the file is a public standard that reputable crawlers follow, not an enforcement mechanism
If your results raise questions, a GEO audit looks at server logs, crawler behaviour and how AI assistants actually describe your brand.
Free Signal Score
Crawlers Can Reach You. Will They Choose You?
Get your free Signal Score to see how your authority, entity data and content compare with the sites AI assistants already cite.
Related services
Turn Access into AI Visibility
FAQ
AI Crawler Questions
How do I check if AI crawlers can access my website?
Enter your domain in the SEOMA AI Crawler Checker. It reads your robots.txt the way crawlers do, following RFC 9309, and requests your homepage as each crawler to detect firewall or CDN blocks. Results show each crawler’s status and the exact rule responsible.
Should I block GPTBot?
GPTBot collects content to train OpenAI models. Blocking it is a reasonable choice if you want to keep your content out of training, and it does not remove you from ChatGPT search, which uses OAI-SearchBot. Most businesses that want to be recommended by ChatGPT keep OAI-SearchBot and ChatGPT-User allowed.
Does blocking Google-Extended affect my Google rankings?
No. Google says Google-Extended does not affect a site’s inclusion in Google Search. It controls whether content may be used to train Gemini models and for grounding in Gemini Apps, so blocking it can reduce visibility in Gemini’s answers.
Why does the checker say a crawler may be blocked when my robots.txt allows it?
Your server refused the request when it identified as that crawler but served a normal browser. That usually points to a CDN, firewall or security plugin rule, such as Cloudflare’s Block AI Bots setting. Confirm it in your provider’s settings, because our test does not come from the crawler’s own IP addresses.
What happens if my robots.txt returns an error?
If robots.txt returns a server error or cannot be reached, crawlers treat the whole site as blocked until it loads again. If it returns 404 Not Found, they treat the site as fully allowed.
Do I need an llms.txt file?
Google does not use llms.txt for Search, so it will not change your Google rankings. Some other AI tools read it, and it is easy to publish, so it can be worth adding once your robots.txt and content are in order.
Is the AI Crawler Checker free?
Yes. It is free and instant, and you do not need to enter an email address.
Free Signal Score
Want AI Assistants to Recommend You?
Book a call and a SEOMA strategist will review your crawler access, Signal Score and competitors, and show you what will move you into AI answers.