AI crawler directory

If blocked: Your content is excluded from OpenAI's model training crawls. It has no effect on whether ChatGPT can cite your site in search.

If allowed: Your public content becomes eligible for inclusion in OpenAI's training data crawls.

If blocked: Your pages become ineligible to appear as a live citation in ChatGPT search answers.

If allowed: Your pages can be fetched and cited in real time when relevant to a user's question.

If blocked: ChatGPT cannot open your page even when a user directly pastes your URL and asks about it.

If allowed: ChatGPT can open and read your page on a user's direct request.

If blocked: Your content is excluded from Anthropic's general crawling.

If allowed: Your public content becomes accessible to Anthropic's crawler.

If blocked: Your pages become ineligible for live citation in Claude's web search.

If allowed: Your pages can be fetched and cited when relevant to a user's question.

If blocked: Claude cannot open your page even on a direct user request.

If allowed: Claude can open and read your page on a user's direct request.

If blocked: Your site drops out of Google Search entirely, and AI features built on search grounding lose access.

If allowed: Your site remains eligible for indexing, ranking and AI-grounded features.

If blocked: Your content is excluded from Gemini/Vertex AI model training while remaining indexable in Search.

If allowed: Your content becomes eligible for use in Gemini and Vertex AI model improvement.

If blocked: Your pages can become unavailable to Bing Search and Copilot experiences that ground answers in Bing's web index.

If allowed: Your pages remain eligible for Bing indexing and related Copilot grounding; it does not guarantee ranking or inclusion in an answer.

If blocked: Your pages become ineligible to be cited in Perplexity's answers.

If allowed: Your pages can be indexed and cited when relevant to a user's question.

If blocked: Amazonbot should not crawl the blocked paths for this general product-improvement and potential model-training purpose.

If allowed: Your public content becomes eligible for Amazonbot's crawl and potential use described by Amazon; use is not guaranteed.

If blocked: The blocked pages become ineligible for Amazon search experiences that depend on this crawler.

If allowed: Your pages remain eligible for discovery in supported Amazon search experiences; visibility is not guaranteed.

If blocked: Robots rules may not fully prevent this user-triggered fetch: Amazon says Amzn-User may not follow all robots.txt directives.

If allowed: Alexa can attempt to retrieve the page for a user's request; successful retrieval or use in a response is not guaranteed.

If blocked: Your blocked pages become unavailable to Mistral's own search index and related discovery experiences.

If allowed: Your public pages remain eligible for Mistral search indexing; inclusion or citation is not guaranteed.

If blocked: Your blocked pages are excluded from future crawls by Mistral's named training agent.

If allowed: Your public content becomes eligible for Mistral's training-data crawl; collection or model use is not guaranteed.

If blocked: Mistral's user-requested fetch should not access blocked paths, which can prevent the page being used for that request.

If allowed: Mistral can attempt to retrieve the page for a user's request; selection or citation is not guaranteed.

If blocked: Your site drops out of Siri and Spotlight results and Apple's other search-powered features.

If allowed: Your site remains eligible for indexing by Apple's search and on-device intelligence features.

If blocked: Your content is excluded from Apple Intelligence model training while remaining eligible for Siri/Spotlight indexing via Applebot.

If allowed: Your content becomes eligible for use in training Apple's foundation models.

If blocked: Your content is excluded from ByteDance's crawl. ByteDance has not published an official crawler policy page, so exact downstream use isn't independently confirmable.

If allowed: Bytespider can fetch your public content. Several third parties have reported it crawling more aggressively and less selectively than other AI crawlers, so allowing it can mean more request volume than other bots.

If blocked: Your content is excluded from future Common Crawl archive snapshots. This is one step removed from any single AI company — it affects a shared dataset multiple labs may draw from, not one platform's crawl directly.

If allowed: Your public content becomes eligible for inclusion in Common Crawl's periodic archive, which is openly downloadable and has historically been used to help pretrain several major language models.