# robots.txt Tester

> Test whether robots.txt allows a URL for Googlebot, GPTBot, ClaudeBot or any crawler you name. See the verdict, the group applied and the rule that decided it.

Source: https://llmstextgenerator.com/robots-txt-tester/
Last updated: 22 October 2026

A free online robots.txt tester, checker and validator. Check whether a path is allowed for a particular crawler, see exactly which rule decided it, and find any line with a syntax problem. Useful when a page has quietly dropped out of search or AI answers and you need to know why.

## How does robots.txt matching work?

The rules are less obvious than they look. The tester follows [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309), the robots.txt standard, and these are the behaviours that cause most of the confusion:

- **Only one group applies.** A crawler uses the group that names it and ignores `User-agent: *` entirely. If `GPTBot` has its own group, nothing in the wildcard group applies to it.
- **Groups for the same crawler are merged.** Two separate `User-agent: Googlebot` groups count as one.
- **The most specific rule wins, not the first.** Where Allow and Disallow both match, the longer pattern wins regardless of order. On a tie, Allow wins.
- **Names are not case-sensitive; paths are.** `googlebot` matches `Googlebot`, but `/Blog` does not match `/blog`.
- **Query strings are part of the path.** `Disallow: /*?sort=` blocks `/shop?sort=price` but not `/shop`.
- **An empty `Disallow:` allows everything.** `Disallow: /` blocks the whole site. The difference is one character.

```
User-agent: *
Disallow: /docs/
Allow: /docs/public/

# /docs/internal    -> blocked  (only Disallow matches)
# /docs/public/faq  -> allowed  (Allow is the longer match)
```

A missing robots.txt (a 404) means no rules, so everything is allowed. A server error is different: the standard tells crawlers to treat a 5xx answer as “disallow everything” until they can read the file, and the tester reports it that way.

## Frequently asked questions

What happened to the robots.txt Tester in Google Search Console?

Google retired it in November 2023 and replaced it with the [robots.txt report](https://support.google.com/webmasters/answer/6062598), under Settings in Search Console. The report shows the robots.txt files Google found for your site, when it last fetched them and any problems, and lets you request a recrawl after a change. It does not test a path against a rule, which is what this tester does, for Googlebot and any other crawler.

The tester says allowed but the page is not indexed.

robots.txt controls crawling, not indexing. A page can be crawlable and still excluded by a `noindex` tag, a canonical pointing elsewhere, or simply not having been picked up yet.

Can I test a site I do not own?

Yes. robots.txt is public by design, and fetching it is an ordinary request for a public file.

Which crawler should I test as?

Whichever one is behaving unexpectedly. Because a named group overrides the wildcard completely, a result for `*` tells you nothing about a crawler that has its own group. “Every group in this file” shows them all at once.