Crawler access
Crawler access checks whether your robots.txt lets crawlers reach the page and points them to a readable sitemap. If robots.txt itself fails to load, crawlers may treat the whole site as off limits until it recovers.
At a glance
- Ladder layer
- P0 Access & eligibility
May and can engines fetch and quote the page? - Where it is measured
- Free check
Run the free check
Reading the result
What good looks like, and what a problem looks like.
Good
robots.txt loads, allows the page and points to a sitemap that can be read.
Not measured
When this cannot be checked, the result says so and leaves it out of the score. Unknown is not zero.
Typical problems
- robots.txt disallows the page.
- robots.txt answers with a server error or not at all.
- No sitemap is referenced, so crawlers discover pages more slowly.
What it cannot prove
- It reads the rules as written. It cannot prove that every crawler follows them.
Does robots.txt keep a page out of search results?
No. robots.txt asks crawlers not to fetch a path, but a blocked page can still be listed if other pages link to it. To keep a page out of results, let it be fetched and mark it noindex.
Where must robots.txt be?
At the root of each host, for example https://example.com/robots.txt. A file on www.example.com does not apply to example.com or to another subdomain.
What happens if robots.txt cannot be fetched?
Under RFC 9309, an answer such as 404 Not Found lets crawlers treat the site as unrestricted, while a server error or an unreachable file can make them treat it as fully disallowed for a time. Either way, fix the file so that the policy is the one you intend.
Sources and further reading
Related dimensions