Robots.txt Checker
& Sitemap Review
Check whether a website’s public discovery files can be reached. Review the robots.txt response, discover same-domain sitemap references and see basic sitemap response and URL-entry details.
Enter a URL from the website
The checker reads only public robots.txt and same-domain sitemap URLs. It does not edit, submit or request access to the website.
How to use this tool
Enter a website URL
Use a public URL on the website you want to review. The tool checks discovery files for that website.
Review robots.txt
Check the returned status and availability. Open the actual file when you need to read its rules.
Inspect the sitemap results
Review each returned sitemap URL, response and basic XML signal. Investigate an unexpected result before changing the website.
Understand the files you are checking
Robots.txt
The file can contain crawling rules and sitemap references. This checker reports its availability and response; it does not simulate every rule for every crawler.
Sitemaps
A sitemap helps describe URLs to search engines. The tool checks public same-domain files and reports basic details from the response.
Your live pages
A reachable sitemap does not prove every listed page is accessible or indexed. Review affected pages separately when investigating discovery problems.
| If you find this | Next step |
|---|---|
| Robots.txt is not found | Open the reported file URL and decide whether the website needs an explicit robots.txt file. |
| A sitemap returns an error | Check that the declared URL is correct and that the sitemap is published and accessible. |
| A sitemap has few or no URL entries | Open the file and check its structure. A sitemap index can reference other sitemap files rather than listing pages directly. |
The tool checks public files and same-domain sitemap URLs. It does not submit sitemaps, edit robots.txt, request Search Console access or confirm Google indexing. Read the actual files when diagnosing a rule or a particular missing page.
Your questions, answered
Does this tool validate every robots.txt rule?
It checks availability and file responses, then reviews sitemap references and basic sitemap details. It does not provide a complete rule-by-rule crawler simulation.
Does it submit my sitemap to Google?
No. It reads public files. Submission and indexing investigation require separate actions through the appropriate website and search tools.
Does a working sitemap guarantee indexing?
No. A sitemap response confirms that a file can be reached in this check. Search engines still assess the listed pages separately.
Why might a sitemap show zero URL entries?
Check the actual file. It may be a sitemap index, contain a different structure or return content that the quick check does not recognise.
Can I use this before a website launch?
The URL must be publicly accessible for the live checker. Keep any deliberate development restrictions in mind when interpreting the response.
What should I use to check an individual page?
Use the Website SEO Checker to review the page response, canonical and robots information alongside its other HTML signals.
Having trouble with crawlability?
Tell us which pages are affected and what the file check returned. We can help investigate the discovery settings and agree the next checks.