Robots.txt Generator
Generate robots.txt rules with a user agent, allowed paths, disallowed paths and sitemap URL.
Workspace
—
How to use this tool
Enter one path per line and optionally provide an absolute sitemap URL, then download robots.txt.
What to know
robots.txt controls crawling, not access or guaranteed indexing. Keep private content behind authentication.
Frequently asked questions
How do I use Robots.txt Generator?
Enter one path per line and optionally provide an absolute sitemap URL, then download robots.txt.
What should I check before using the result?
robots.txt controls crawling, not access or guaranteed indexing. Keep private content behind authentication.
Is an account required?
No. This tool is free and the calculation takes place in your browser.