Robots.txt Generator
Create a properly formatted robots.txt file, telling search engine crawlers exactly which parts of your site they can and cannot access.
User-agent: * Disallow: /admin/ Disallow: /private/ Sitemap: https://example.com/sitemap.xml
About the Robots.txt Generator
A robots.txt generator creates a properly formatted robots.txt file, telling search engine crawlers which parts of your site they can and cannot access.
How to use
1
Specify which paths or sections you want to allow or disallow.
2
Add any sitemap URL you want crawlers to discover.
3
Copy the generated file and upload it to your site's root directory.
Frequently asked questions
Where does robots.txt need to be placed?
It must be placed at the root of your domain, such as yoursite.com/robots.txt, for search engines to find it.
Does robots.txt prevent a page from appearing in search results?
Not necessarily. It blocks crawling, but a page can still be indexed if it's linked from elsewhere, so use a noindex meta tag if you need to prevent indexing specifically.
Explore more tools in the SEO category.