Robots.txt Generator

Generate a robots.txt file for crawl rules and sitemaps.

Features

User-agent rules
Sitemap line helpers
Copy-ready robots.txt

How to use robots.txt

1
Set user-agent and path rules.
2
Add sitemap URLs if you have them.
3
Copy robots.txt to your site root.

Examples

Block admin

Input

Disallow: /admin

Output

User-agent: *
Disallow: /admin

About robots.txt

Robots.txt Generator scaffolds Allow/Disallow rules and sitemap lines for a site you are launching or restructuring. Review the output against Search Console after deploy - generators cannot know every bot edge case.

Frequently asked questions

No. It is a crawl hint, not access control.

Major crawlers generally respect it; malicious bots may not.

Yes - you still upload the file to your host.

Related tools

HTTP & API Tools

New

Sitemap Generator

Build an XML sitemap from a list of URLs or paths.

Meta Tag Generator

Generate SEO and social meta tags for title, description, and Open Graph.