Robots.txt Generator
Generate robots.txt files for crawler control. User-agent presets (Googlebot, Bingbot, etc.), allow/disallow rules, wildcard patterns, crawl-delay, sitemap directive, comments, and validation warnings. 100% client-side.
About Robots.txt Generator
Generate robots.txt files for crawler control. User-agent presets (Googlebot, Bingbot, etc.), allow/disallow rules, wildcard patterns, crawl-delay, sitemap directive, comments, and validation warnings. 100% client-side. Everything runs locally in your browser — your data never leaves your device.
How to use
- Enter your input in the tool above.
- Adjust any options to your preference.
- Use the Copy or Download buttons to save the result.
- Everything happens locally — your data never leaves your browser.
FAQ
What is robots.txt?
robots.txt is a text file at the root of your site (e.g. example.com/robots.txt) that tells search engine crawlers which parts of your site they may or may not crawl. It uses the Robots Exclusion Protocol.
What's the difference between Disallow and noindex?
Disallow tells crawlers not to crawl a URL — but if a URL is linked from elsewhere, it may still appear in search results without a description. To prevent indexing entirely, use the robots meta tag with noindex instead.
What are wildcard patterns?
The * wildcard matches any sequence of characters and $ marks the end of a URL. For example, Disallow: /*? blocks all URLs with a query string, and Disallow: /*.pdf$ blocks all PDF files.
What extra features does this tool have compared to others?
(1) User-agent presets — Googlebot, Bingbot, Slurp, DuckDuckBot, Baiduspider, YandexBot, facebookexternalhit, AhrefsBot, SemrushBot, GPTBot. (2) Allow/disallow toggle per rule. (3) Path pattern builder with wildcard support. (4) Sitemap auto-include. (5) Crawl-delay directive. (6) Comments. (7) Live preview. (8) Validation warnings (e.g., path without leading slash). (9) Stats — rule count per user-agent. (10) History (localStorage, last 20).
Is my data sent anywhere?
No. robots.txt generation is pure string templating in your browser. History is stored in localStorage on this device only.
Related tools
Affiliate Commission Calculator
Calculate affiliate commissions across four program types — flat percent, tiered percent (progressive brackets), recurring (monthly × N months), and hybrid (initial + recurring). Apply refund-rate and tax-withholding adjustments, compute effective commission rate, compare two program types side-by-side, and project break-even sales to hit a target income. History (localStorage) + shareable URL. 100% client-side.
Affiliate Link Cloaker & Redirect Generator
Cloak affiliate links into pretty URLs and generate redirect scripts (PHP, JavaScript, HTML meta refresh), plus .htaccess, nginx, and robots.txt rules. CSV import of slug/affiliate_url pairs, slug auto-generator from hostname, click-tracking parameter builder, URL-shortener compatibility check, history (localStorage), shareable URL. 100% client-side.
Anchor Text Distribution Analyzer
Analyze anchor text distribution from backlink data. Categorize anchors as exact/partial match, branded, generic, naked URL, or image. Compute distribution %, warn on over-optimization, and export. 100% client-side.