Robots.txt Generator
Build a robots.txt file visually with user-agent groups, allow/disallow rules, sitemaps, and more.
About This Tool
The Robots.txt Generator is a free browser-based tool that helps
webmasters and SEO professionals create correctly formatted
robots.txt files without memorizing the syntax. A
robots.txt file lives at the root of your website and
tells web crawlers which parts of your site they are allowed or
forbidden to access. It is one of the oldest and most fundamental
mechanisms of the Robots Exclusion Protocol, first introduced in
1994.
Search engine crawlers like Googlebot, Bingbot, and others check
for a robots.txt file before crawling your site. By
specifying rules per user-agent, you can control how different
crawlers interact with your pages. For example, you might allow
Google full access while blocking AI training crawlers like GPTBot
or CCBot. You can also set crawl-delay directives to reduce server
load from aggressive bots.
This tool supports all standard robots.txt
directives: User-agent, Allow,
Disallow, Crawl-delay,
Sitemap, and Host. The visual editor
lets you add multiple user-agent groups, each with their own set
of rules. A live preview updates as you build, so you can see the
exact output that will go into your file.
All processing happens entirely in your browser. No data is sent to any server, making this tool safe for any project. The built-in validator checks for common mistakes such as missing user-agents, invalid paths, conflicting rules, and malformed sitemap URLs, helping you catch errors before deploying.
How to Use
- Optionally select a preset template from the dropdown to start with a common configuration (Allow All, Block All, Block AI Crawlers, Standard Blog, or E-commerce).
- Add or edit user-agent groups. Each group specifies a crawler name (e.g. Googlebot, *, GPTBot) and a set of Allow/Disallow rules.
- For each group, add Allow or Disallow rules with the path you want to control. The path input provides suggestions for common paths like
/admin,/api, and/wp-admin. - Set an optional Crawl-delay per group to tell crawlers how many seconds to wait between requests.
- Add one or more Sitemap URLs to help crawlers discover your sitemap files.
- Optionally set a Host directive if you need to specify a preferred domain (used by Yandex).
- Review the live preview on the right to verify the output looks correct.
- Click Validate to check for common issues, then Copy or Download the file.
FAQ
What is a robots.txt file?
A robots.txt file is a plain text file placed at the root of a website (e.g. https://example.com/robots.txt) that instructs web crawlers which pages or sections of the site they are allowed or forbidden to access. It follows the Robots Exclusion Protocol standard.
Is robots.txt mandatory for SEO?
No, a robots.txt file is not mandatory, but it is highly recommended. Without one, crawlers will attempt to access all pages on your site. A well-configured robots.txt helps you manage crawl budget, prevent indexing of private areas, and keep duplicate or low-value pages out of search results.
Does robots.txt block pages from appearing in Google?
Not exactly. A Disallow rule prevents crawlers from fetching a page, but if other sites link to that URL, Google may still index the URL (without content) and show it in search results. To fully prevent indexing, use a noindex meta tag or HTTP header in combination with robots.txt.
Can I block AI crawlers with robots.txt?
Yes. Many AI companies respect robots.txt directives. You can add rules for user-agents like GPTBot, ChatGPT-User, Google-Extended, CCBot, and anthropic-ai with Disallow: / to block them. Use the "Block AI Crawlers" preset to get started quickly.
What is the Crawl-delay directive?
The Crawl-delay directive tells crawlers to wait a specified number of seconds between successive requests. This can help reduce server load from aggressive crawlers. Note that Googlebot does not support Crawl-delay -- use Google Search Console to adjust Googlebot's crawl rate instead. Bingbot and Yandex do respect this directive.
Where should I place the robots.txt file?
The file must be placed at the root of your domain, accessible at https://yourdomain.com/robots.txt. It must be served with a text/plain content type. Placing it in a subdirectory will not work -- crawlers only check the root URL.
Is my data safe with this tool?
Yes. The entire tool runs client-side in your browser. No data is transmitted to any server. You can verify this by checking your browser's network tab -- there are zero outbound requests when generating or copying your robots.txt file.
Related Tools
Meta Tag Generator
Generate HTML meta tags, Open Graph, and Twitter Card tags with live SNS preview.
URL Encode/Decode
Encode and decode URLs, parse query parameters, and build query strings.
HTTP Status Codes
Browse, search, and learn about all HTTP status codes with detailed explanations.
Text Sort & Deduplicate
Sort lines alphabetically, remove duplicates, trim whitespace, and filter empty lines instantly.