Crawler Permissions

Create and validate robots.txt files for search engine crawling control