robots.txt Generator
Build a correct robots.txt file with simple options - rules for all or specific user agents, disallow paths, sitemap URLs and host - and copy/download the result.
Updated September 2026 · Written and verified by Akash Pandey
robots.txt Generator
How to use the robots.txt generator
- Add allow / disallow rules as paths (or whole paths with wildcards).
- Optionally target a specific user agent and set a crawl delay.
- Add your sitemap URL(s) and optional host.
- Click Generate, then copy or download
robots.txt.
Frequently asked questions
What does robots.txt do?
It tells compliant search-engine crawlers which URL paths they may or may not visit. It is a crawl policy, not a security control.
Where do I place the file?
At the root of the site, e.g. https://example.com/robots.txt. Subdomain content is governed by that subdomain's own robots.txt.
How do I allow one path but block the rest?
Block the whole site with "Disallow: /" for the agent, then list "Allow:" lines above the disallow for the paths you want crawled. Order matters for some crawlers.
Should I include the sitemap line?
Yes - a "Sitemap:" line points crawlers at your sitemap even if it is not linked elsewhere. Most sites list one here.
Is robots.txt the same as noindex?
No. robots.txt stops crawling but Google may still index the URL from other signals. A meta noindex removes the page from results while letting it be crawled.
Can the generator harm my site?
Blocking critical assets (CSS/JS) can hurt rendering quality signals. The generator labels common mistakes and keeps the output to standard directives.
Where should the robots.txt file be uploaded on a website?
The robots.txt file must be placed in the root directory of your website domain, for example: https://example.com/robots.txt.
How do I specify a sitemap location inside robots.txt?
Add a Sitemap: directive followed by the full absolute URL to your sitemap (e.g., Sitemap: https://example.com/sitemap.xml).