Bytespider checker
Find what stops AI reading your site — and get your full report with the exact fixes.
ChatGPT
Claude
Gemini
PerplexityInstant preview · Full report free by email
Includes all AI platforms and all five readiness categories.
canairead.me
Bytespider checker · Full AI readiness report included
Example AI readiness report
canairead.me · 92/100
What we check
Bytespider is ByteDance's crawler, believed to gather content for search and AI model training on behalf of the company behind TikTok. Unlike OpenAI or Anthropic, ByteDance hasn't published an official crawler policy page, and several site owners have reported it crawling more heavily and less selectively than other AI bots — which is why many choose to block it explicitly rather than leave it to a default rule.
- Whether Bytespider is explicitly allowed or disallowed in robots.txt
- Whether a wildcard rule unintentionally affects it
- Whether your robots.txt has any rule for it at all
Passing technical checks is only the starting point. Read why your site might not show up in ChatGPT to work through access, rendering and content issues before expecting citations.
Crawlers involved
Bytespider
ByteDance
ByteDance's crawler (the company behind TikTok), gathering content believed to support search and AI model training.
Common problems
No explicit rule
Most robots.txt files say nothing about Bytespider specifically, meaning it defaults to allowed.
Assumed blocked, actually allowed
Some site owners believe a generic security plugin catches it, then find on inspection that it doesn't.
How to fix them
Add an explicit directive
Write `User-agent: Bytespider` followed by `Allow: /` or `Disallow: /` so the decision is intentional rather than left to whatever your host or plugin defaults to.
Watch server load, not just robots.txt
robots.txt is a request most reputable crawlers honour, not an enforcement mechanism — if load is a concern, pair it with rate limiting or a firewall rule.
Frequently asked
Does Bytespider ignore robots.txt?
There's no independently verifiable evidence either way, since ByteDance hasn't published an official crawler policy. Some site owners have anecdotally reported continued requests after disallowing it; treat robots.txt as a request that reputable crawlers typically honour, not a technical guarantee.
Is Bytespider the same as TikTok's app?
No. Bytespider is a separate, automated web crawler operated by ByteDance (TikTok's parent company) — it has nothing to do with how the TikTok app itself works.
Should I block Bytespider?
That depends on whether you're comfortable with ByteDance's crawler accessing your content and how much request volume you're willing to absorb. There's no universally correct answer — it's the same training-data-opt-out decision as GPTBot, just from a different company.