# robots.txt examples > Tested robots.txt examples: allow all, disallow all, block a folder, block AI training but keep AI search, add a sitemap, slow Bingbot, the WordPress default. Source: https://llmstextgenerator.com/robots-txt-examples/ Last updated: 22 October 2026 Copy-ready files for the jobs people search for most. Each one was run through our robots.txt tester before it went on this page, and the notes say what it does and what it does not. **A robots.txt is a list of groups: one or more `User-agent` lines, then the rules for those crawlers.** A crawler follows only the most specific group that names it, so a crawler with its own group ignores the `User-agent: *` rules ([Google’s robots.txt specification](https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec)). The file always lives at `https://yourdomain.com/robots.txt`. Test any of these with your own addresses in the [robots.txt tester](https://llmstextgenerator.com/robots-txt-tester/), or build your own in the [robots.txt generator](https://llmstextgenerator.com/robots-txt-generator/). ## Allow all crawlers robots.txt ``` User-agent: * Allow: / ``` Every crawler may read every page. An empty `Disallow:` line does the same, since a rule without a path is ignored ([Google’s robots.txt specification](https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec)). Having no robots.txt at all also works: Google treats a 404 as if no file existed and assumes no restrictions, as the standard allows ([RFC 9309](https://www.rfc-editor.org/rfc/rfc9309)). ## Disallow all crawlers robots.txt ``` User-agent: * Disallow: / ``` Asks every crawler to stay out of the whole site. Two things to know before you use it: - **It does not remove pages from Google.** Google can still list an address it finds through links, without reading it. To keep pages out of results, let them be crawled and use `noindex` ([Google’s noindex documentation](https://developers.google.com/search/docs/crawling-indexing/block-indexing)). - **It is not access control.** Well-behaved crawlers comply; anything else may not. For a staging site, use a password instead. ## Block one folder, with an exception robots.txt ``` User-agent: * Disallow: /admin/ Allow: /admin/help/ ``` Blocks `/admin/` but keeps `/admin/help/` open. When an Allow and a Disallow both match, Google uses the rule with the longest path, and the least restrictive one if they are equally long ([Google’s robots.txt specification](https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec)). ## Block AI training, keep AI search robots.txt ``` User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: CCBot User-agent: Meta-ExternalAgent Disallow: / User-agent: * Allow: / ``` Asks the crawlers that collect training data to stay away, while the crawlers that index pages for AI answers, such as `OAI-SearchBot`, `Claude-SearchBot` and `PerplexityBot`, still follow the `*` group and may read everything. Each operator documents which name does which job: [OpenAI’s crawler documentation](https://developers.openai.com/api/docs/bots), [Anthropic’s crawler documentation](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler), [Perplexity’s crawler documentation](https://docs.perplexity.ai/docs/resources/perplexity-crawlers) and [Google’s crawler list](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers). `Google-Extended` controls use of your content by Gemini and has no effect on Google Search. ## Block all AI crawlers, including AI search robots.txt ``` User-agent: GPTBot User-agent: OAI-SearchBot User-agent: ClaudeBot User-agent: Claude-SearchBot User-agent: PerplexityBot User-agent: Google-Extended Disallow: / User-agent: * Allow: / ``` This also takes your pages out of the answers those assistants give with citations. Search engines like Googlebot are unaffected. The [generator](https://llmstextgenerator.com/robots-txt-generator/) has this and the previous example as presets, with the full list of crawler names it knows. ## Point crawlers to your sitemap robots.txt ``` User-agent: * Allow: / Sitemap: https://example.com/sitemap.xml ``` The `Sitemap` line must be a full address, including `https://` and the host ([Google’s robots.txt specification](https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec)). It can appear anywhere in the file, outside any group, and you can list several. Our [XML sitemap generator](https://llmstextgenerator.com/xml-sitemap-generator/) makes the file it points to. ## Slow down Bingbot robots.txt ``` User-agent: bingbot Crawl-delay: 5 User-agent: * Allow: / ``` `Crawl-delay` asks for a pause in seconds between requests. Bing honours it ([Bing’s note on Crawl-delay](https://blogs.bing.com/webmaster/2012/05/03/to-crawl-or-not-to-crawl-that-is-bingbots-question)). Google does not support it and sets its own pace from how your server responds ([Google’s robots.txt specification](https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec)), and Yandex stopped using it in 2018 ([Yandex’s Crawl-delay page](https://yandex.com/support/webmaster/robot-workings/crawl-delay.html)). ## The WordPress default, explained What WordPress serves until you add your own ``` User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Sitemap: https://example.com/wp-sitemap.xml ``` WordPress serves this file itself until a real one exists. The first lines come from [WordPress’s do_robots() reference](https://developer.wordpress.org/reference/functions/do_robots/): they keep crawlers out of the admin area but allow `admin-ajax.php`, which some themes call from public pages. The paths follow your admin address, so they differ if that has moved. The `Sitemap` line was added in WordPress 5.5 by [WP_Sitemaps::add_robots()](https://developer.wordpress.org/reference/classes/wp_sitemaps/add_robots/), and only appears while the site is public. Since WordPress 5.3, ticking “Discourage search engines” no longer adds `Disallow: /`; WordPress uses a robots meta tag instead ([WordPress’s do_robots() reference](https://developer.wordpress.org/reference/functions/do_robots/)). **Test before you publish.** One stray `Disallow: /` can hide a whole site. Paste your file into the [robots.txt tester](https://llmstextgenerator.com/robots-txt-tester/) and try the addresses that matter, for each crawler you care about. ## Frequently asked questions Does Disallow remove a page from Google? No. A blocked address can still be listed from links. Use `noindex` on a crawlable page instead ([Google’s noindex documentation](https://developers.google.com/search/docs/crawling-indexing/block-indexing)). Do I need a robots.txt at all? No. With no file, crawlers assume they may read everything. You need one only to restrict crawlers or to point them to your sitemap. What happens if my robots.txt returns a server error? Google pauses crawling while it keeps retrying the file, then falls back to rules it saw before ([Google’s robots.txt specification](https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec)). The standard says crawlers must treat an unreachable file as a full disallow ([RFC 9309](https://www.rfc-editor.org/rfc/rfc9309)). Keep it reliably available.