Robots.txt Checker
A misconfigured robots.txt can accidentally block Google from crawling your entire site. This is one of the most common and catastrophic SEO mistakes. Validate yours right now.
What does the Robots.txt Checker check?
Everything you need to know about your robots.txt checker in one report.
File Detection
Checks that robots.txt exists and is accessible at the correct URL.
Syntax Validation
Validates the format of all User-agent, Allow, and Disallow directives.
Blocks All Crawlers
Flags the critical error of Disallow: / for all bots — a site-wide crawl block.
Blocked Assets
Identifies CSS, JavaScript, or image files that are blocked but needed for rendering.
Sitemap Directive
Checks that your robots.txt references your sitemap URL for easy discovery.
Crawl Delay
Detects Crawl-delay settings that may slow down Google's indexing of your site.
How it works
Enter your domain
Paste your website URL — SEO-Snap fetches your robots.txt automatically.
Rule analysis
We parse every directive and check for common misconfigurations and best practice violations.
Review and fix
Get a report of issues with severity levels — critical errors vs. warnings.
Ready to check your site?
Use the Robots.txt Checker free — no account required for a basic check. Sign up for full history, PDF reports, and all 12 tools at once.
Frequently asked questions
What is robots.txt?
robots.txt is a file at the root of your website that tells search engine crawlers which pages they can and cannot visit. It's the first file Google checks when crawling your site.
Can robots.txt stop Google from indexing pages?
robots.txt blocks crawling, not indexing. Google can still index a blocked URL if another site links to it — they just won't see the content. To prevent indexing, use a noindex meta tag instead.
What happens if I have Disallow: / in robots.txt?
This blocks all crawlers from accessing your entire site — a critical SEO error. Google will eventually de-index all your pages. This is often set accidentally during development.
Should I block my /admin or /dashboard pages?
Yes. Add Disallow: /admin/ and Disallow: /dashboard/ to prevent crawlers from wasting their crawl budget on pages that don't need to be indexed.
More free SEO tools
Page Speed Checker
Test your website speed & Core Web Vitals instantly
Meta Tag Checker
Analyse your title, description, OG tags and more
Broken Link Checker
Find all dead links on your website for free
Schema Markup Validator
Validate your structured data and rich results eligibility