Create a correctly formatted robots.txt file for your website, block AI training bots, and control what search engines can crawl. Free and instant.
robots.txt to your site's root folder, so it loads at yoursite.com/robots.txt.Loading...
Free Online Robots.txt Generator – Control How Crawlers See Your Site
A robots.txt file is a small plain-text file placed at the root of your website that tells search engine crawlers and other bots which parts of your site they may visit and which parts they should skip. It is the first thing a well-behaved crawler checks before it starts browsing your pages. Writing this file by hand is simple in theory but easy to get subtly wrong, and a single mistake — like an accidental "Disallow: /" — can hide your entire site from search results. Our free robots.txt generator builds the file for you, correctly formatted every time.
Start from a ready-made preset, such as Allow Everything, Block Everything, or the Blogger preset, or build your own rules from scratch. Add the paths you want to keep away from crawlers, such as admin folders, internal search pages, or duplicate content, using the quick-add buttons for common paths. With growing concern over AI companies scraping content to train their models, this tool also includes one-click checkboxes to block AI crawlers like GPTBot, ClaudeBot, Google-Extended, CCBot, and PerplexityBot, along with SEO scraper bots like AhrefsBot and SemrushBot, while still allowing normal search engines like Google and Bing to index your site.
Why Your Site Needs a Robots.txt File
Without a robots.txt file, crawlers will generally try to access every page on your site, which can waste crawl budget on low-value pages like admin panels or search results, leaving your important content crawled less often. A well-configured file directs crawlers toward the pages that matter and away from the ones that don't. It gives you a simple, standard way to signal which AI bots you don't want scraping your content for training data. It also lets you point crawlers directly to your sitemap, ensuring new posts are discovered quickly. And it can prevent accidental indexing of internal pages, staging areas, and files that were never meant to be public, which protects both your SEO and your visitors' privacy.
Understanding User-agent, Disallow, and Allow Rules
Every rule in a robots.txt file is built around three directives. User-agent identifies which crawler the rule applies to — using "*" means the rule applies to all crawlers, while naming a specific bot (like GPTBot) creates an exception just for that crawler. Disallow tells the crawler not to visit a path, and Allow carves out an exception inside a disallowed folder — this is why you often see both together, for example disallowing /wp-admin/ but allowing /wp-admin/admin-ajax.php, since WordPress needs that one file to be reachable. Paths are matched by prefix: "Disallow: /search" blocks /search, /search?q=hello, and /search-results, all at once. Wildcards are supported by Google and Bing — "/*?*" blocks all URLs that contain a query string, which is useful for stopping duplicate-content crawl traps. One important limit: robots.txt only asks well-behaved crawlers to stay away. It does not password-protect anything, and it does not remove already-indexed pages from search results — for that, you need a "noindex" meta tag on the page itself, or password protection. And because the file must sit at your domain root to be found, on Blogger you do not upload it at all: you paste the code into Settings → Crawlers and indexing → Custom robots.txt, and Blogger serves it automatically at yoursite.com/robots.txt.
Free, Instant, and No Signup
This generator runs entirely in your browser, updates the code live as you build your rules, and requires no account or installation. Once you're happy with the result, copy it or download it as a ready-to-use robots.txt file. It works on Windows, Mac, Android, and iPhone, with no limit on how many times you generate or how complex your rules are.