Robots.txt Validator

Why Validate Your robots.txt?

A single typo in your robots.txt can accidentally block your entire site from search engines or leave sensitive pages exposed to crawlers. Common mistakes include missing User-agent directives, Disallow rules before any User-agent, malformed Sitemap URLs, and unrecognised directives. Our validator catches all of these issues with clear error messages.

Features

• Remote URL validation — fetch and validate robots.txt from any domain.
• File upload — upload a local robots.txt file for instant checking.
• Paste directly — paste raw content for quick validation.
• Directive status badges — at-a-glance indicators for User-agent, Disallow, Allow, and Sitemap.
• Error & warning separation — critical errors shown in red, recommendations in yellow.
• Line-level feedback — errors reference the specific line number.

How It Works

The validator parses each non-comment line of your robots.txt and checks directive ordering, required fields, and value formats. User-agent must appear before any Allow or Disallow rules. Sitemap URLs are validated as absolute URLs. Crawl-delay values must be non-negative numbers. Any line that doesn't match a known directive is flagged as a warning.

What Gets Checked

• At least one User-agent directive is present.
• Disallow/Allow directives only appear after a User-agent.
• Sitemap URLs are well-formed absolute URLs.
• Crawl-delay values are valid non-negative numbers.
• Unrecognised directives are flagged as warnings.