robots.txt Generator
Build robots.txt files visually with checkbox-based controls. Select user-agents like Googlebot and Bingbot, configure allow and disallow paths, add sitemap URLs, and set crawl-delay. Preview, copy, or download the generated file.
Preview
User-agent: *
How to Use the robots.txt Generator
- Select a user-agent (use * for all bots, or pick specific crawlers)
- Choose paths to disallow by clicking the path buttons or adding custom paths
- Optionally add allow exceptions for disallowed directories
- Enter your sitemap URL so search engines can discover your pages
- Copy or download the generated robots.txt file
What Is robots.txt?
The robots.txt file is a plain text file placed at the root of a website that tells web crawlers which pages or sections of the site they are allowed or disallowed from accessing. It follows the Robots Exclusion Protocol, a standard that virtually all major search engine crawlers respect.
While robots.txt does not prevent pages from being indexed (use the noindex meta tag for that), it controls crawl budget by directing bots away from unimportant areas like admin panels, staging content, or duplicate pages. A well-configured robots.txt helps search engines focus their crawling on the pages that matter most for your site's SEO.
Common Use Cases
Controlling Crawl Budget
Direct search engine bots away from low-value pages like admin areas, search result pages, and tag archives so they spend their crawl budget on important content.
Blocking Sensitive Areas
Prevent crawlers from accessing staging environments, development endpoints, or internal API routes that should not appear in search results.
Managing AI Crawlers
Block AI training crawlers like GPTBot and CCBot from scraping your content while still allowing search engine bots to index your pages.
Sitemap Discovery
Include your sitemap URL in robots.txt so search engines can discover and crawl all your pages efficiently, even without submitting the sitemap manually.
robots.txt Best Practices
Always Include a Sitemap
Adding a Sitemap directive to your robots.txt helps search engines discover all your pages, especially new or deeply nested content.
Test Before Deploying
Use Google Search Console's robots.txt Tester to verify your rules work as intended before uploading the file to your server.
Don't Block CSS and JS
Search engines need access to CSS and JavaScript files to render and understand your pages correctly. Blocking them can hurt your rankings.
Use Specific Rules Over Broad Blocks
Instead of blocking entire directories, use specific path patterns to disallow only the pages you want to exclude. Overly broad rules can accidentally block important content.