Robots.txt Generator

User-agent: *
Disallow: /admin
Disallow: /private
Allow: /

Why Use a robots.txt?

The robots.txt file sits at the root of your website and tells search engine crawlers which pages or sections they are allowed or forbidden to access. A properly configured robots.txt prevents crawlers from indexing admin panels, staging pages, duplicate content, and other areas you want to keep out of search results. Our visual editor makes it easy to build one without memorising the syntax.

Features

  • Visual rule builder — add user-agent groups and paths through a form, no syntax to remember.
  • Multiple user-agent groups — create separate rules for Googlebot, Bingbot, or any crawler.
  • Allow & Disallow paths — add as many paths as you need per group.
  • Sitemap directive — include your sitemap URL so crawlers discover it automatically.
  • Crawl-delay support — optionally slow down aggressive bots.
  • Live preview — see the generated robots.txt update as you edit.
  • Download or copy — save as robots.txt or copy to your clipboard.

How It Works

Each user-agent group you create becomes a block in the output. The generator follows the Robots Exclusion Protocol: User-agent lines come first, followed by Disallow and Allow directives, then an optional Crawl-delay. Sitemap directives are placed at the end of the file, outside any group, so all crawlers can find them.

Best Practices

  • Always include a User-agent: * group as a catch-all for unrecognised bots.
  • Block admin, staging, and internal search result pages to avoid thin content penalties.
  • Never block CSS or JavaScript files — Google needs them for rendering.
  • Always include a Sitemap directive pointing to your XML sitemap.

Tags