Set your crawl rules below and get a ready-to-upload robots.txt file.
Generated robots.txt
Choose whether to allow or block all crawlers by default, add any paths you want to disallow, add your sitemap URL, and the generated text appears instantly. Copy it into a file named exactly robots.txt and upload it to your site's root folder.
No — a browser can't reliably fetch another website's files directly due to cross-origin security restrictions. This tool generates a new robots.txt for you to review and upload yourself; to check a live site's existing file, visit yoursite.com/robots.txt directly in your browser.
It must be placed at the root of your domain, so it's accessible at yoursite.com/robots.txt exactly — not in a subfolder.
Need something else? See the full tools homepage.