How to test robots.txt
- Enter a website domain or direct robots.txt URL.
- Retrieve the file.
- Review user-agent groups and directives.
- Enter a URL path and select a user agent when rule testing is supported.
- Review the matched rule and any declared sitemap locations.
What is robots.txt?
robots.txt is a text file placed at the root of a host to communicate crawling rules to compliant crawlers. It can identify user-agent groups and provide allow or disallow paths.
Robots.txt is not an access-control system. A blocked URL can still be publicly accessible, and sensitive information should never rely on robots rules for protection.
What should a robots.txt checker show?
A useful report may include:
- Whether the file was found
- HTTP status and final URL
- Parsed user-agent groups
- Allow and disallow directives
- Matching rule for a tested path
- Sitemap declarations
- Syntax notes
- Lines the parser could not interpret
The checker should show the observed rule and explain the decision. It should not simply display a score.
How robots.txt rule testing works
Rule testing compares a selected crawler and path with the applicable group and directives. The exact matching behavior must follow the parser implemented by the product.
The result should identify:
- The selected user agent
- The tested URL or path
- The group applied
- The most relevant matching rule
- The resulting allow or disallow interpretation
A test result does not prove that a crawler visited the URL or that a search engine indexed it.
Robots.txt is not noindex
Blocking crawling does not reliably remove a URL from search results. If a URL must not be indexed, use an appropriate indexability method that the crawler can access and process, and remove the URL from public discovery when necessary.
Use the HTTP Header Checker to inspect X-Robots-Tag values. Use the SEO Checker to review other page-level signals.
Sitemap declarations
A robots.txt file can declare one or more sitemap URLs. The checker should list the declarations it finds without claiming that the sitemaps are valid.
Use the XML Sitemap Checker to retrieve and validate a declared sitemap separately.
Common robots.txt issues
- File placed at the wrong path
- Rules written for the wrong host or protocol
- Important sections disallowed unintentionally
- Incorrect user-agent grouping
- Conflicting or misunderstood allow/disallow patterns
- Sitemap URL pointing to an unavailable location
- Using robots.txt to hide private information
- Blocking resources needed to render or understand pages
When a missing robots.txt file is acceptable
A website is not required to have a robots.txt file. If no special crawl rules are needed, a missing file may be acceptable. The tool should distinguish “not found” from “site failure.”
Frequently asked questions
Does robots.txt prevent indexing?
Not reliably. It primarily controls crawling for compliant bots. A URL can still be discovered through links or other sources.
Can robots.txt protect private content?
No. Use authentication and proper access control for private information.
Does every website need robots.txt?
No. A site with no special crawl directives may operate without one.
Can this tool confirm how Googlebot will behave?
It can test the rules using the implemented parser, but live crawler behavior and indexation require external verification.
Does a sitemap line mean the sitemap is valid?
No. It only declares a location. Validate the file with the XML Sitemap Checker.