Robots.txt Generator
Create a clear robots.txt file with allow, disallow and sitemap directives.
Create a robots.txt starting point
Build user-agent, allow, disallow and sitemap directives with explicit line-by-line output.
What robots.txt controls
A robots.txt file gives compatible web crawlers instructions about paths they may or may not crawl. It normally lives at the root of a host, such as https://example.com/robots.txt.
How to use this generator
- Choose the crawler user-agent, or use
*for a general rule group. - Add allow and disallow paths one per line.
- Add your absolute sitemap URL if applicable.
- Review every path before deploying the file at the site root.
Robots.txt is not access control
A disallow rule does not make a URL private. Sensitive information must be protected with real authentication and authorization. Also remember that blocking crawling and preventing indexing are different concepts.
Test carefully before broad blocks
A rule such as Disallow: / can prevent a compliant crawler from accessing the whole site for that user-agent. On production sites, review generated rules with someone who understands the site's crawl and indexing requirements.