Robots.txt Generator
Build a valid robots.txt with allow/disallow paths, sitemap, and user-agent targeting. Preview updates live as you type.
Rules
robots.txt Preview
Save this as robots.txt at your site's root, e.g. https://example.com/robots.txt.
Why use our Robots.txt Generator?
No Syntax Guesswork
Fill in a simple form and get correctly formatted User-agent, Allow, Disallow, and Sitemap directives.
Live Preview
See the generated robots.txt update instantly as you edit paths, so you can review before publishing.
Copy and Deploy
Copy the result straight into a robots.txt file at your site's root — no server round trip required.
What is a Robots.txt Generator?
robots.txt is a plain-text file placed at the root of a website (for example, https://example.com/robots.txt) that tells well-behaved search engine crawlers which paths they may or may not crawl, and optionally points them to your sitemap. It uses simple directives like User-agent, Allow, Disallow, and Sitemap.
Our free robots.txt generator builds this file from a short form — set a user-agent, list paths to allow or disallow, and add your sitemap URL. The tool assembles valid syntax instantly in your browser so you can copy it straight into your project without memorizing the exact directive format.
What You Can Configure
- User-agent:Target a specific crawler (like Googlebot) or use * to apply rules to all crawlers.
- Allow paths:List paths crawlers are explicitly permitted to access, one per line.
- Disallow paths:List paths crawlers should avoid, such as /admin or /api, one per line.
- Sitemap URL:Point crawlers to your sitemap.xml so they can discover pages more efficiently.
- Crawl-delay:Optionally suggest a delay in seconds between requests for supporting crawlers.
- Live preview:See the exact file contents update as you type, then copy with one click.
Common Use Cases
Launching a New Site
Generate a starter robots.txt before going live so crawlers know which paths are safe to index.
Blocking Admin & Staging Paths
Disallow paths like /admin, /wp-admin, or /staging so they don't show up in search results.
Pointing to a Sitemap
Add a Sitemap directive so search engines can discover your full sitemap.xml automatically.
Per-Crawler Rules
Target a specific bot like Googlebot or Bingbot when you need different rules than your default policy.
E-commerce Filtering
Disallow faceted search or filter URL patterns that create near-duplicate content for crawlers.
Quick SEO Audits
Draft an updated robots.txt while reviewing a site's crawl budget and indexation issues.
How to Generate a Robots.txt File
Set the User-Agent
Use * for all crawlers, or specify a bot name like Googlebot.
Add Allow/Disallow Paths
List one path per line for paths to allow or block from crawling.
Add a Sitemap URL
Enter your sitemap.xml URL so crawlers can find it automatically.
Copy and Publish
Copy the generated text and save it as robots.txt at your site's root.
Benefits of a Well-Formed Robots.txt
Correct Syntax
Avoid typos in directive names that could cause crawlers to ignore your rules.
Better Crawl Budget
Keep crawlers focused on important pages instead of admin or duplicate paths.
Faster Discovery
A Sitemap directive helps search engines find and index new pages sooner.
Consistent Across Bots
Easily add per-crawler rules without hand-writing repetitive blocks.
No Server Needed
Generate and preview the file entirely client-side before deploying it.
Free Forever
No signup, no limits, and no watermark in the generated output.
Robots.txt Tips
- •It's advisory, not security: robots.txt asks well-behaved crawlers not to access paths — it does not restrict access or hide sensitive data.
- •Use absolute sitemap URLs: Sitemap directives should use a full URL, not a relative path.
- •Test after publishing: Use Google Search Console's robots.txt tester to confirm your rules behave as expected.
- •Don't block CSS/JS: Blocking asset paths crawlers need to render your page can hurt how search engines see your content.
- •One file per site: robots.txt must live at the root domain — subdirectory robots.txt files are ignored.
- •Keep it simple: Prefer a small number of clear rules over dozens of overlapping Allow/Disallow lines.
Generate Your Robots.txt Now
Whether you're launching a new site, blocking admin paths, or pointing crawlers to your sitemap, our free generator builds correct robots.txt syntax from a simple form — right in your browser.
Fill in the fields above, review the live preview, and copy the result into your project. No signup, no server, and no limits on how many times you can regenerate it.
Related tools
Frequently Asked Questions
Where do I put the generated robots.txt file?
Save it as robots.txt and upload it to the root of your domain, for example https://example.com/robots.txt. It must be at the root — subdirectory files are ignored by crawlers.
Does robots.txt prevent pages from being indexed?
Not reliably on its own. Disallow tells crawlers not to fetch a path, but a page can sometimes still be indexed by URL if it's linked elsewhere. Use a noindex meta tag or header for pages you truly want excluded from search results.
Is robots.txt a security measure?
No. robots.txt is a public, advisory file that any well-behaved crawler can read. Never rely on it to hide sensitive paths — use authentication or server-side access controls instead.
What does User-agent: * mean?
The asterisk (*) is a wildcard that matches all crawlers. Rules under User-agent: * apply to any bot that doesn't have its own specific block.
Can I have multiple Disallow lines?
Yes. List each path you want to block on its own line, one per Disallow directive, under the relevant User-agent block.
Do all search engines respect Crawl-delay?
No. Crawl-delay is not part of the original robots.txt standard and is only honored by some crawlers (notably Bing). Major engines like Google generally ignore it.
Is this robots.txt generator free?
Yes. It's completely free to use with no signup and no limit on how many files you can generate.
Is my data uploaded when I use this tool?
No. The form fields and generated file are processed entirely in your browser. Nothing is sent to a server.