Etoolio

Robots.txt Generator

Free robots.txt generator. Block folders, keep GPTBot and other AI bots out of training but in AI search, add your sitemap, and test any page before uploading.

Updated

1. Start from

Every crawler may crawl the whole site. Add paths to block below.

2. Paths for all crawlers

Nothing blocked: every crawler may crawl every page.

A path blocks everything that starts with it: /admin blocks /admin, /admin/ and /administrator. Use * for any characters and $ for the end, as in /*.pdf$. Allow re-opens part of a blocked folder; the longer rule wins.

3. AI crawlers

Blocking training keeps your pages out of new AI models. Blocking AI search also removes you from ChatGPT, Claude and Perplexity answers. Googlebot and Bingbot are not here: they power Google and Bing search, including AI Overviews and Copilot, so blocking them removes you from search.

AI training
AI search (cite and link to you)
AI assistants fetching a page for a user

4. Rules for one crawler (optional)

A crawler with its own group reads only that group and skips the rules for all crawlers. Keep “Also apply the rules for all crawlers” ticked unless you mean to give it different access.

5. Sitemap and crawl delay

Google ignores Crawl-delay; Bing and some AI crawlers honour it. Leave it empty unless crawlers overload your server.

Your robots.txt

User-agent: *
Allow: /

Upload it to the root of your site so it loads at https://yoursite.com/robots.txt. Each subdomain needs its own.

Check a page

GooglebotAllowedAllow: /
BingbotAllowedAllow: /
GPTBotAllowedAllow: /
OAI-SearchBotAllowedAllow: /
ClaudeBotAllowedAllow: /
Claude-SearchBotAllowedAllow: /
PerplexityBotAllowedAllow: /
Google-ExtendedAllowedAllow: /

Checked with the same rules as our robots.txt tester: the longest matching rule wins, and Allow wins a tie.

Make a robots.txt that does what you think it does

A robots.txt file is a few lines of text, but a small slip can hide a whole site from Google or let a crawler into pages you meant to block. This generator writes the file from your choices, following the published standard (RFC 9309) and Google's documented rules, and checks the result as you go: type any page into "Check a page" and it shows which crawlers may fetch it and which line decides.

How to use it

  • Start from a preset: allow everything, WordPress's own default, or block everything for a staging site.
  • Add paths to block for all crawlers, such as /admin/ or /cart/, and Allow rules to re-open part of a blocked folder.
  • Choose an AI policy: allow all AI bots, block AI training but stay in AI search, or block them all. Fine-tune bot by bot.
  • Optionally give one crawler its own rules, add your sitemap URL and a crawl delay.
  • Copy or download the file and upload it so it loads at https://yoursite.com/robots.txt.

What goes in a robots.txt file

User-agent: *
Disallow: /admin/
Allow: /admin/help/

User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /

Sitemap: https://example.com/sitemap.xml
  • User-agent names the crawler a group is for. * means every crawler that has no group of its own. Several User-agent lines in a row share the rules below them.
  • Disallow blocks every URL that starts with the path. Disallow: /admin blocks /admin/ and also /administrator; add the slash to block only the folder.
  • Allow re-opens a path inside a blocked one. When rules conflict, the longer path wins, and Allow wins a tie. The order of lines does not matter.
  • * matches any characters and a final $ marks the end of the URL, so Disallow: /*.pdf$ blocks every PDF.
  • Sitemap gives the full URL of your sitemap. It applies to all crawlers, wherever it sits in the file.
  • Crawl-delay asks a crawler to wait between requests. Bing and some AI crawlers honour it; Google ignores it.

The mistake that catches most sites

A crawler obeys only the most specific group that names it. Add a User-agent: Googlebot group and Googlebot stops reading the User-agent: * group entirely, so the /admin/ block you wrote for everyone no longer applies to it. That is why the generator copies the rules for all crawlers into any crawler's own group unless you untick "Also apply the rules for all crawlers". For the same reason, AI bots you allow are left out of the file: unnamed, they follow your general rules, while a group saying Allow: / would let them past them.

AI crawlers: training, search and assistants

AI companies now run separate bots for separate jobs, so you can opt out of one without the others. Blocking a training bot keeps your pages out of future models. Blocking a search bot removes you from that company's AI answers, which also means no citations or links back.

TokenCompanyWhat blocking it does
GPTBotOpenAIKeeps your pages out of OpenAI model training
OAI-SearchBotOpenAIRemoves you from ChatGPT search answers
ChatGPT-UserOpenAIAsks ChatGPT not to open your pages for users; OpenAI says robots.txt may not apply
ClaudeBotAnthropicKeeps your pages out of Claude model training
Claude-SearchBotAnthropicRemoves you from Claude's search results
Claude-UserAnthropicStops Claude opening your pages when a user asks
Google-ExtendedGoogleStops Gemini training and grounding on your pages; Google Search is unaffected
Applebot-ExtendedAppleStops Apple AI training; Siri and Spotlight search are unaffected
PerplexityBotPerplexityRemoves you from Perplexity answers
Perplexity-UserPerplexityLittle: Perplexity says this fetcher generally ignores robots.txt
meta-externalagentMetaKeeps your pages out of Meta AI training
meta-webindexerMetaRemoves you from Meta AI search answers
meta-externalfetcherMetaAsks Meta AI not to open your links for users; Meta says it may bypass robots.txt
CCBotCommon CrawlKeeps your pages out of the open web archive many models train on
BytespiderByteDanceKeeps your pages out of ByteDance AI crawling
AmazonbotAmazonKeeps your pages out of Amazon's crawl, which may train Amazon AI

Googlebot and Bingbot are not in this list on purpose. They power Google and Bing search, including Google's AI Overviews and Microsoft Copilot answers, so blocking them removes you from search altogether.

Common robots.txt examples

  • Allow everything: User-agent: * then Allow: /. An empty Disallow: means the same.
  • Block everything (staging only): User-agent: * then Disallow: /. Protect the site with a password too.
  • WordPress default: Disallow: /wp-admin/ plus Allow: /wp-admin/admin-ajax.php, and the sitemap WordPress serves at /wp-sitemap.xml (or your SEO plugin's sitemap).
  • Block internal search results: Disallow: /search, or Disallow: /*?s= on WordPress.

What robots.txt cannot do

  • It does not remove pages from Google. A blocked page can still be indexed from links. Use a noindex meta tag or X-Robots-Tag header, and leave the page crawlable so Google can see it.
  • It is not security. The file is public and voluntary. Never list secret URLs in it; protect them with a login.
  • Do not block CSS or JavaScript. Google renders pages like a browser; blocking /wp-content/, /_next/ or *.js can hurt how your pages rank. The generator warns you.
  • Keep it reachable. If robots.txt returns a server error, Google stops crawling your site until it can read the file. A missing file (404) just means "crawl everything".

Sources

Related tools

Check a live site's file with the robots.txt tester, see which AI crawlers a domain allows with the AI crawler checker, add opt-out meta tags and headers with the AI training opt-out kit, and give AI models a map of your content with an llms.txt file.

Good to know

Robots.txt Generator, frequently asked questions

What is a robots.txt file?

robots.txt is a plain text file at the root of a website, such as https://example.com/robots.txt, that tells crawlers which URLs they may fetch. It is organised in groups: a User-agent line names the crawler, and Disallow and Allow lines list the paths it may not or may crawl. Search engines and the major AI companies follow it, but it is a request, not a lock.

How do I create a robots.txt file?

Pick a starting point (allow everything, WordPress, or block everything for a staging site), add any paths you want to block, choose how AI crawlers should be treated and add your sitemap URL. The generator writes the file as you go and shows which crawlers can reach any page you test. Copy or download it and upload it to the root of your site.

Where do I put robots.txt?

In the top-level folder of the host it applies to, so it loads at https://example.com/robots.txt. A file in a subfolder is ignored. Each subdomain, such as shop.example.com, needs its own file, and the file must be plain UTF-8 text.

How do I block AI crawlers with robots.txt?

Name each AI crawler in a User-agent line and give it Disallow: /. To stay out of training but still appear in AI answers, block the training bots (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent, CCBot and others) and leave the search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) allowed. The generator's middle AI option does exactly that.

Does robots.txt stop a page appearing in Google?

No. robots.txt controls crawling, not indexing. A blocked page can still be indexed, usually without a description, if other sites link to it. To keep a page out of Google, let it be crawled and add a noindex meta tag or X-Robots-Tag header instead.

Should I add my sitemap to robots.txt?

Yes. A Sitemap line with the full URL, such as Sitemap: https://example.com/sitemap.xml, helps Google, Bing and other crawlers find it without you submitting it to each one. You can list several sitemaps, and the line applies to all crawlers wherever it sits in the file.

Is a robots.txt file required?

No. If a site has no robots.txt (the server returns 404), crawlers assume they may crawl everything. Most sites still benefit from one, to point crawlers at the sitemap, keep them out of admin or cart pages and set an AI policy. A robots.txt that returns a server error is worse than none: Google then stops crawling the site.

Is the robots.txt generator free and private?

Yes. It is free with no sign-up, and everything runs in your browser. Nothing you type is uploaded.

Keep going

Related tools

Robots.txt TesterTest any URL against a site's robots.txt for Googlebot, Bingbot, GPTBot and more, and see the exact rule that decides.AI Crawler Access CheckerCheck if AI crawlers like GPTBot and ClaudeBot can access your site.AI Training Opt-Out KitStop AI companies training on your site, robots.txt, meta tags & opt-out links.llms.txt GeneratorGenerate an llms.txt file for your site so AI models can read it.