Robots.txt Generator
Generate a correct robots.txt file that tells search engines which paths they may crawl. Add allow and disallow rules, a crawl delay and your sitemap URL, then copy the finished file to the root of your site.
What this means
robots.txt is a plain-text file at your site root that politely asks crawlers which areas to skip, such as admin pages or private folders. It is not a security boundary — it only affects cooperating bots.
How we calculate it
Formula
Format: User-agent: name, then Allow/Disallow lines per rule, then Sitemap: URL. An asterisk matches every bot.Worked example
Block admin pages from being indexed.
- 1Enter the paths to block, like /admin.
- 2Add your sitemap URL.
- 3Copy robots.txt to your site root.
Important assumptions
- Sitemap URLs should be absolute, starting with https://.
Frequently asked questions
Does robots.txt block indexing?
It prevents crawling, not indexing. A page linked from elsewhere can still appear; for strict control use the noindex meta tag too.
What does the asterisk mean?
User-agent: * applies your rules to all search engine bots.
Related tools
Meta Tag Generator
Generate complete meta and Open Graph tags for your page.
XML Sitemap Generator
Generate an XML sitemap from your URLs.
Schema Markup Generator
Generate JSON-LD schema for articles, products and businesses.
QR Code Generator
Create a downloadable QR code for any URL or text, with colors and size.
Password Generator
Create strong, random passwords with adjustable length and character sets.
Lorem Ipsum Generator
Generate placeholder paragraphs, sentences or words for mockups.