robots.txt AI checker
Understand which AI bots your robots.txt allows, with your content and search foundations in the same report.
ChatGPT
Claude
Gemini
PerplexityInstant preview · Full report free by email
Includes all AI platforms and all five readiness categories.
canairead.me
robots.txt AI checker · Full AI readiness report included
Example AI readiness report
canairead.me · 92/100
Fix → publish → rescan
A robots.txt decision should be intentional
Common accidental configuration
User-agent: *
Disallow: /
# Left over from stagingThe wildcard group blocks the whole site for every compliant crawler. A CDN block can cause the same outcome without appearing in this file.
Example: allow search, decline training
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /This is an example, not a universal recommendation. Preserve private-path exclusions and verify the published result with the focused report above.
Read the homepage-only scope and matching limitations, then compare individual OAI-SearchBot and GPTBot decisions.
What we check
robots.txt is still the single most common reason AI systems can't reach a website — usually a broad rule that blocks more than intended, not a deliberate choice. We parse yours against every major AI crawler at once.
- Explicit rules for OpenAI, Anthropic, Amazon, Mistral, Google, Microsoft and Perplexity agents
- Wildcard rules that may unintentionally match AI crawlers
- Sitemap directive presence and validity
- Syntax errors that could cause rules to be ignored
Passing technical checks is only the starting point. Read why your site might not show up in ChatGPT to work through access, rendering and content issues before expecting citations.
See exactly what is tested and what a score cannot prove in the scan methodology and limitations.
Common problems
Copy-pasted template rules
A robots.txt copied from another site blocks crawlers the original owner didn't want, for reasons that don't apply here.
Disallow: / left over from staging
A blanket block meant for a staging environment was never removed after launch.
How to fix them
Write rules per crawler, not by wildcard
Explicit User-agent blocks for each AI crawler you care about remove ambiguity entirely.
Validate after every deploy
robots.txt changes are easy to make by accident — check it after CMS migrations or platform switches.
Frequently asked
Where should robots.txt live?
At the root of your domain — yoursite.com/robots.txt. It won't be respected anywhere else.
Can robots.txt guarantee a crawler won't visit?
It's a request most reputable crawlers honour, including the major AI companies' bots, but it isn't a technical enforcement mechanism on its own.