Robots.txt

Build robots.txt files with user-agent rules, allow/disallow paths, and sitemap directives