WebUtils Logo

Robots.txt Generator

Build a robots.txt file from editable crawler rule groups. Configure user agents, allowed and disallowed paths, optional crawl delays, and sitemap URLs, then copy or download the result.

Features

Preset templatesMultiple user agentsAllow/Disallow rulesCrawl delay settingsSitemap referencesCommon path patterns

How to use the Robots.txt Generator

  1. 1Start from a preset or create a user-agent group.
  2. 2Add allow and disallow path rules for that crawler.
  3. 3Add crawl-delay or sitemap values only when they apply.
  4. 4Review the generated file carefully, then copy or download robots.txt.

Common uses

  • Block crawling of private or duplicate route patterns.
  • Declare one or more XML sitemap locations.
  • Create separate rules for specific crawlers.

Robots.txt is not access control

Compliant crawlers may avoid disallowed paths, but the URLs remain publicly accessible and can still appear in search results without content. Protect sensitive resources with authentication and use page-level noindex where appropriate.

Frequently asked questions

Can robots.txt keep private data secure?

No. Use authentication and authorization. Robots rules are public crawl requests, not security controls.

Where does the file go?

Serve it at the root of the relevant host, such as https://example.com/robots.txt.

Should I block CSS and JavaScript assets?

Usually no. Search engines often need page assets to render and evaluate the page correctly.

Related Code & Development Tools

Sitemap Generator

Create XML sitemaps for better SEO

Meta Tags & Hashtags Generator

Generate SEO meta tags and social media hashtags from content

Open Graph Tags Generator

Generate Open Graph meta tags for social media previews

Web Manifest Generator

Create manifest.json for Progressive Web Apps