Web Indexing Control

Validate robots.txt syntax, directives, and crawler permissions