Validate Robots.txt

Validate robots.txt syntax, directives, and crawler permissions