Skip to content
clkr

Loading tools…

    Robots.txt Generator

    Generate a robots.txt file with this free online robots.txt generator: disallow paths, block AI crawlers, add sitemaps, and test URLs against the rules.

    Loading…

    What next?

    Found it useful? Share Robots.txt Generator

    Free, no sign-up, works in the browser — send it to a friend or add it to your site.

    Related tools

    Part of the toolkit

    How to use Robots.txt Generator

    1. 1Choose whether search engines may crawl your site, then list paths to disallow and any allow exceptions (or pick a preset)
    2. 2Optionally block AI crawlers, add your sitemap URL and a crawl-delay
    3. 3Test URLs against the rules, then copy or download robots.txt and upload it to your domain root

    Frequently asked questions

    Where do I put robots.txt?▾

    In the root of your domain, so it loads at https://yoursite.com/robots.txt. Crawlers don't look for it in subfolders, and each subdomain needs its own file.

    How do I block AI crawlers like GPTBot?▾

    Add a group for their user agents with Disallow: /. Tick 'Block AI crawlers' to add GPTBot, ClaudeBot, Google-Extended, CCBot, PerplexityBot and others while search engines keep full access.

    Which rule wins when Allow and Disallow both match?▾

    Google and Bing use the most specific rule — the longest matching path. If an Allow and a Disallow are the same length, Allow wins. The built-in tester applies exactly these rules.

    Does robots.txt remove pages from Google?▾

    No. It stops crawling, but a blocked URL can still be indexed if other sites link to it. To keep a page out of search results use a noindex meta tag and let it be crawled.

    Embed Robots.txt Generator on your website▾

    Paste this snippet into your page. The embedded tool runs in your visitor's browser and shows a small “Powered by clkr.work” link.