AI crawler access
AI crawler access checks whether the crawlers that AI search tools rely on, such as OAI-SearchBot, PerplexityBot and Claude-SearchBot, are allowed to fetch your page and actually receive it. An assistant cannot cite a page its crawler was turned away from.
At a glance
- Ladder layer
- P0 Access & eligibility
May and can engines fetch and quote the page? - Where it is measured
- Free check
Run the free check
Reading the result
What good looks like, and what a problem looks like.
Good
Search crawlers are allowed by your robots.txt and receive the same page your visitors see.
Not measured
When this cannot be checked, the result says so and leaves it out of the score. Unknown is not zero.
Typical problems
- robots.txt disallows a search crawler that assistants rely on.
- A firewall or CDN refuses or challenges AI crawlers while ordinary visitors get through.
- Crawlers receive different content from what visitors see.
What it cannot prove
- Our tests show how your site answers a crawler's name. They do not prove that the provider itself visited.
- A refusal is reported only when it repeats while an ordinary browser request succeeds before and after. A one-off refusal is shown as inconsistent, not as a block.
- Blocking training crawlers such as GPTBot is a business choice. It is reported as your policy, not as a defect.
Should I block GPTBot?
It is a business choice. GPTBot and ClaudeBot collect text to train AI models. Blocking them does not stop the search crawlers, such as OAI-SearchBot and Claude-SearchBot, that fetch pages for answers that cite their sources. We report training access as your policy, not as a defect.
Why are AI crawlers refused when my robots.txt allows them?
A firewall, a CDN or a bot-protection service can refuse a request before robots.txt is considered. robots.txt states a policy; your server decides what each request receives. The check compares a request under each crawler name with an ordinary browser request.
Does a test under a crawler name prove that the crawler can read my site?
No. It shows how your site answers that name when we send the request. The provider’s own crawler comes from its own addresses, which some services verify. Your server logs show real visits.
Related dimensions