robots.txt AI checker
Understand which AI bots your robots.txt allows, with your content and search foundations in the same report.
ChatGPT
Claude
Gemini
PerplexityInstant preview · Full report free by email
Includes all AI platforms and all five readiness categories.
canairead.me
robots.txt AI checker · Full AI readiness report included
Example AI readiness report
canairead.me · 92/100
What we check
robots.txt is still the single most common reason AI systems can't reach a website — usually a broad rule that blocks more than intended, not a deliberate choice. We parse yours against every major AI crawler at once.
- Explicit rules for GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot and Google-Extended
- Wildcard rules that may unintentionally match AI crawlers
- Sitemap directive presence and validity
- Syntax errors that could cause rules to be ignored
Common problems
Copy-pasted template rules
A robots.txt copied from another site blocks crawlers the original owner didn't want, for reasons that don't apply here.
Disallow: / left over from staging
A blanket block meant for a staging environment was never removed after launch.
How to fix them
Write rules per crawler, not by wildcard
Explicit User-agent blocks for each AI crawler you care about remove ambiguity entirely.
Validate after every deploy
robots.txt changes are easy to make by accident — check it after CMS migrations or platform switches.
Frequently asked
Where should robots.txt live?
At the root of your domain — yoursite.com/robots.txt. It won't be respected anywhere else.
Can robots.txt guarantee a crawler won't visit?
It's a request most reputable crawlers honour, including the major AI companies' bots, but it isn't a technical enforcement mechanism on its own.