Robots.txt Tester: Check if a URL Is Blocked for Googlebot & AI Bots

Test any URL against robots.txt, the way Google reads it

robots.txt is a short text file at the root of a website that tells crawlers which URLs they may fetch. It looks simple, but small details decide the outcome: which group of rules a crawler picks, which of several matching rules wins, and what a stray character does. This tester applies the published standard (RFC 9309) and Google's documented behaviour to any URL and any crawler, and shows you the exact line that decides.

Google retired its own robots.txt Tester in Search Console in late 2023. The robots.txt report that replaced it shows whether Google could fetch the file, but it does not answer "is this URL blocked for this crawler?". That is the question this tool answers.

How to use it

Fetch from a website: enter a domain or a full page URL. The tool reads that host's live /robots.txt, reports the HTTP status and any redirects, and tests the page you typed.
Paste or edit: paste a robots.txt, or edit the fetched one, and the results update as you type. Use this to check a change before you deploy it.
Test several URLs at once, one per line, as full URLs or paths such as /search?q=shoes.
Pick a crawler: Googlebot, Googlebot-Image, AdsBot-Google, Bingbot, Applebot, GPTBot, ClaudeBot, PerplexityBot and more, or type a custom user-agent.

How crawlers decide

1. Pick the group

A group starts with one or more User-agent lines. A crawler uses the group that names its own token, matched without regard to case, and only falls back to User-agent: * when no group names it. If several groups name the same crawler, their rules are combined. Some crawlers have special behaviour: Googlebot-Image and Googlebot-News fall back to the Googlebot group, and AdsBot-Google ignores * entirely, so it only obeys a group that names it.

Only the product token counts. User-agent: Googlebot/2.1 means Googlebot, but User-agent: Googlebot Image also means plain Googlebot, because matching stops at the space. The tester warns you when that happens.

2. Find the most specific rule

Within the group, every Allow and Disallow whose pattern matches the start of the URL path is a candidate. The longest pattern wins. When an Allow and a Disallow are exactly as long, the less restrictive one (Allow) wins. The order of lines in the file makes no difference.

Rules	URL	Result
`Disallow: /admin/` `Allow: /admin/help`	`/admin/help/faq`	Allowed: the Allow pattern is longer
`Disallow: /*.pdf$`	`/files/report.pdf`	Blocked
`Disallow: /*.pdf$`	`/files/report.pdf?v=2`	Allowed: $ anchors the end
`Disallow: /page` `Allow: /page`	`/page`	Allowed: a tie goes to Allow
`Disallow:` (empty)	anything	Allowed: an empty rule blocks nothing

3. Wildcards

* matches any run of characters, including none. A $ at the very end of a pattern means "the URL ends here". Matching is case-sensitive, and the query string counts as part of the path, so Disallow: /*?sort= blocks sorted listings.

What the HTTP status means

200: the rules in the file apply.
404 or another 4xx: there is no robots.txt, so everything may be crawled.
5xx, 429 or no answer: Google treats the whole site as blocked and stops crawling until robots.txt responds properly. A broken robots.txt is one of the fastest ways to lose crawling.
Redirects: Google follows up to five. After that it treats robots.txt as missing.
Size: Google reads only the first 500 KiB. Rules after that are ignored.

Common mistakes the tester flags

A path that does not start with /, which never matches.
Rules placed before any User-agent line, which belong to no group.
Directives Google ignores: Noindex (unsupported since 2019), Host (an old Yandex directive) and Crawl-delay (Bing and Yandex use it, Google does not).
Relative sitemap paths. Sitemap needs a full URL.
Serving an HTML page at /robots.txt, which leaves the site with no rules at all.

Blocking is not the same as hiding

robots.txt controls crawling, not indexing. A blocked URL can still appear in search results, usually without a description, if other sites link to it. To keep a page out of Google, let it be crawled and add a noindex meta tag or X-Robots-Tag header. Also remember that robots.txt is public and voluntary: well-behaved crawlers follow it, but it is not access control.

Related tools

To see at a glance which AI crawlers your site allows, use the AI crawler checker. To see which bots actually visit, upload your access log to the server log analyzer. To give AI models a map of your content, build an llms.txt file.

Robots.txt Tester, frequently asked questions

What does the robots.txt tester do?

It reads a robots.txt (fetched live from any website, or pasted in) and tells you whether a crawler such as Googlebot, Bingbot, GPTBot or ClaudeBot may crawl a URL. It shows the exact line that decides, flags lines crawlers ignore, and lists every common crawler's result for the same URL.

Didn't Google have its own robots.txt tester?

Google retired the robots.txt Tester in Search Console in late 2023 and replaced it with a robots.txt report that shows fetch status, not per-URL answers. This tool fills that gap: it applies Google's documented matching rules to any URL and any crawler.

Which rule wins when Allow and Disallow both match?

The most specific rule wins, meaning the one with the longest path pattern. If an Allow and a Disallow are exactly the same length, Google uses the less restrictive one, so Allow wins. The order of lines in the file does not matter.

How do * and $ work in robots.txt?

An asterisk matches any run of characters, so Disallow: /*.pdf blocks any URL containing .pdf. A dollar sign at the end anchors the match to the end of the URL, so Disallow: /*.pdf$ blocks /report.pdf but not /report.pdf?download=1.

Does blocking a page in robots.txt remove it from Google?

No. robots.txt controls crawling, not indexing. A blocked URL can still be indexed, usually without a description, if other pages link to it. To keep a page out of search results, allow crawling and add a noindex meta tag or X-Robots-Tag header.

What happens if robots.txt returns an error?

If robots.txt returns 404 or another 4xx error, crawlers treat the site as having no rules and may crawl everything. If it returns a 5xx server error, Google treats the whole site as blocked and stops crawling until it gets a good response. The tester shows which case applies.

Is it free and private?

Yes. It is free with no sign-up. Fetching only reads the public robots.txt file, and all parsing and matching happen in your browser, so a robots.txt you paste or edit is never uploaded.