LLMsTextGenerator
Free tool

robots.txt Tester

A free online robots.txt tester, checker and validator. Check whether a path is allowed for a particular crawler, see exactly which rule decided it, and find any line with a syntax problem. Useful when a page has quietly dropped out of search or AI answers and you need to know why.

Test a path

Last updated

Test a robots.txt rule

Fetch a live file or paste one, then test a path for a crawler. You see the verdict, the group that applied and the exact rule that decided it.

Query strings count: /shop?sort=price is tested as typed.

Result

How does robots.txt matching work?

The rules are less obvious than they look. The tester follows RFC 9309, the robots.txt standard, and these are the behaviours that cause most of the confusion:

  • Only one group applies. A crawler uses the group that names it and ignores User-agent: * entirely. If GPTBot has its own group, nothing in the wildcard group applies to it.
  • Groups for the same crawler are merged. Two separate User-agent: Googlebot groups count as one.
  • The most specific rule wins, not the first. Where Allow and Disallow both match, the longer pattern wins regardless of order. On a tie, Allow wins.
  • Names are not case-sensitive; paths are. googlebot matches Googlebot, but /Blog does not match /blog.
  • Query strings are part of the path. Disallow: /*?sort= blocks /shop?sort=price but not /shop.
  • An empty Disallow: allows everything. Disallow: / blocks the whole site. The difference is one character.
User-agent: *
Disallow: /docs/
Allow: /docs/public/

# /docs/internal    -> blocked  (only Disallow matches)
# /docs/public/faq  -> allowed  (Allow is the longer match)

A missing robots.txt (a 404) means no rules, so everything is allowed. A server error is different: the standard tells crawlers to treat a 5xx answer as “disallow everything” until they can read the file, and the tester reports it that way.

Frequently asked questions

What happened to the robots.txt Tester in Google Search Console?

Google retired it in November 2023 and replaced it with the robots.txt report, under Settings in Search Console. The report shows the robots.txt files Google found for your site, when it last fetched them and any problems, and lets you request a recrawl after a change. It does not test a path against a rule, which is what this tester does, for Googlebot and any other crawler.

The tester says allowed but the page is not indexed.

robots.txt controls crawling, not indexing. A page can be crawlable and still excluded by a noindex tag, a canonical pointing elsewhere, or simply not having been picked up yet.

Can I test a site I do not own?

Yes. robots.txt is public by design, and fetching it is an ordinary request for a public file.

Which crawler should I test as?

Whichever one is behaving unexpectedly. Because a named group overrides the wildcard completely, a result for * tells you nothing about a crawler that has its own group. “Every group in this file” shows them all at once.