Robots.txt Analysis

Validate robots.txt syntax, directives, and crawler permissions