robots.txt Generator
A free robots.txt creator with AI crawler presets. Decide which crawlers may read your site, including the AI crawlers that index pages for answers and the ones that collect training data, and get a correctly formatted file. Everything runs in your browser.
Last updated
Build a robots.txt
Pick a starting point, adjust individual crawlers, and copy the file. Everything runs in your browser.
Your robots.txt
Which AI crawlers can robots.txt control?
AI companies run separate crawlers for separate jobs, and each one has its own User-agent name. That is what lets you allow one job and refuse another.
- Search and answer crawlers index pages so an assistant can cite them. Blocking them takes you out of those answers.
- Training crawlers collect pages that may be used to train future models. Blocking them has no effect on search or on AI answers.
- User-triggered fetchers load a single page because a person asked about it. Some operators say robots.txt may not apply to these, because a human started the request.
Here is every AI crawler the generator knows about. Names and robots.txt behaviour were checked against each operator’s documentation on 16 October 2026.
| User agent | Operator | Kind | What it does | Obeys robots.txt |
|---|---|---|---|---|
OAI-SearchBot | OpenAI | AI search and answers | Indexes pages for ChatGPT search results and citations. | Yes |
Claude-SearchBot | Anthropic | AI search and answers | Indexes pages to improve Claude search results. | Yes |
PerplexityBot | Perplexity | AI search and answers | Indexes pages for Perplexity answers. Not used for training. | Yes |
DuckAssistBot | DuckDuckGo | AI search and answers | Fetches pages for DuckAssist answers. | Yes |
Google-CloudVertexBot | AI search and answers | Crawls sites whose owners asked Google to build Vertex AI agents from them. | Yes | |
GPTBot | OpenAI | AI training | Crawls pages that may be used to train OpenAI models. | Yes |
ClaudeBot | Anthropic | AI training | Crawls pages that may be used to train Claude models. | Yes |
Google-Extended | AI training | A control token, not a crawler: decides whether Gemini may use pages Googlebot fetched. No effect on Google Search. | Yes | |
Applebot-Extended | Apple | AI training | A control token: decides whether Apple may use pages Applebot fetched to train its models. | Yes |
Meta-ExternalAgent | Meta | AI training | Crawls pages for training AI models and indexing. | Yes |
CCBot | Common Crawl | AI training | Builds the open Common Crawl archive, widely used as training data. | Yes |
Bytespider | ByteDance | AI training | Crawls pages for ByteDance models. | Yes |
Amazonbot | Amazon | AI training | Crawls pages for Amazon services, including Alexa answers and model training. | Yes |
cohere-ai | Cohere | AI training | Collects pages for Cohere models. | Yes |
MistralAI-Training | Mistral AI | AI training | Crawls pages to build datasets for training Mistral models. | Yes |
ChatGPT-User | OpenAI | User-triggered AI fetchers | Fetches a page when a ChatGPT user asks about it. | Not always |
Claude-User | Anthropic | User-triggered AI fetchers | Fetches a page when a Claude user asks about it. | Yes |
Perplexity-User | Perplexity | User-triggered AI fetchers | Fetches a page when a Perplexity user asks about it. | Not always |
Meta-ExternalFetcher | Meta | User-triggered AI fetchers | Fetches individual links a Meta AI user asked about. | Not always |
MistralAI-User | Mistral AI | User-triggered AI fetchers | Visits a page when a Vibe user asks a question, and links to it as a source. Not used for automatic crawling or training. | Yes |
Google-Agent | User-triggered AI fetchers | Agents on Google infrastructure that browse and act when a user asks. Google lists no robots.txt token for it. | Not always | |
Google-GeminiNotebook | User-triggered AI fetchers | Fetches pages a Gemini Notebook user added as sources (formerly Google-NotebookLM). | Not always |
robots.txt is a request, not a lock. Well-behaved crawlers honour it. Anything determined to ignore it will. If content must not be read, put it behind a login rather than a disallow rule.
What do the presets do?
| Preset | AI search | AI training | User-triggered | Search engines |
|---|---|---|---|---|
| Allow AI search, block training | Allow | Block | Allow | Default |
| Block all AI crawlers | Block | Block | Block | Default |
| Allow everything | Default | Default | Default | Default |
“Default” means the crawler follows your rules for everyone. A crawler set to Allow gets its own group. Because a crawler with its own group ignores the User-agent: * rules entirely, the generator copies your blocked paths into that group, so allowing a crawler never quietly opens your /admin/ folder to it.
One rule catches people out: Google-Extended controls whether Gemini may use your content, and has no effect on Google Search. Blocking it does not remove you from search results. Blocking Googlebot does, and the generator warns you before you do it.
Where does robots.txt go on your platform?
robots.txt must load at the root of your domain, at https://yourdomain.com/robots.txt. A copy in a sub-folder is ignored, and each subdomain needs its own file. How you put it there depends on your platform:
| Platform | Where it goes |
|---|---|
| Most hosting (cPanel, Apache, Nginx) | Upload the file to the folder your domain serves from, often public_html/. |
| WordPress | WordPress serves a virtual robots.txt until a real one exists. Edit it in Yoast SEO (Tools, then File editor) or Rank Math (General Settings, then Edit robots.txt), or upload a file to the site root. |
| Shopify | Add a robots.txt.liquid template to your theme (Online Store, Themes, Edit code). Shopify builds the file from it. |
| Next.js | Return the rules from app/robots.ts, or put a static file in public/robots.txt. |
| Netlify, Vercel and other static hosts | Put the file in the folder that gets published, often public/ or the build output. |
| Webflow | Paste the rules into the robots.txt field in the site’s SEO settings. |
| Wix | Use the robots.txt editor in Wix’s SEO settings. |
| Squarespace | Squarespace writes the file itself and you cannot edit it. Its settings can ask known AI crawlers to stay away. |
| Blogger | In your blog’s settings, under Crawlers and indexing, turn on Enable custom robots.txt and paste the rules. They replace Blogger’s default file. |
After uploading, open the address in a private window to check it loads as plain text, then run it through the robots.txt tester.
robots.txt or llms.txt?
They do different jobs, and many sites use both. robots.txt tells crawlers which URLs they may fetch. llms.txt is an optional, hand-picked index of your most useful pages for AI tools to read. A crawler you block in robots.txt should not fetch the pages your llms.txt lists, so keep the two consistent.
| robots.txt | llms.txt | |
|---|---|---|
| Answers | Which paths may this crawler fetch? | Which pages matter, and what is on them? |
| Read by | Search engines and AI crawlers | AI tools and agents that look for it |
| Status | A standard (RFC 9309) | A proposal; Google Search does not need it |
The longer comparison covers sitemap.xml too, and the llms.txt generator builds the other file.
Frequently asked questions
Does robots.txt actually stop bots?
Only the ones that choose to follow it. The standard itself says its rules “are not a form of access authorization” (RFC 9309). Well-behaved crawlers follow them; some user-triggered AI fetchers and badly behaved bots do not. If content must stay private, protect it with a login or server rules, not robots.txt.
Does blocking a page in robots.txt remove it from Google?
No. Google can still list a blocked URL it finds through links, just without reading the page. To keep a page out of Google, let it be crawled and add a noindex meta tag or X-Robots-Tag header. If you block the page as well, Google never sees the noindex (Google’s noindex documentation).
Can one file have rules for several crawlers?
Yes, one group per User-agent. A crawler follows only the most specific group that names it and ignores the User-agent: * group, so rules you want everywhere must be repeated in its own group (Google’s robots.txt specification). The generator does this for you.
Does Google follow Crawl-delay?
No. Google does not support Crawl-delay (Google’s robots.txt specification) and sets its own pace from how quickly your server responds. Bing honours it (Bing’s note on Crawl-delay), and Yandex stopped using it in February 2018 in favour of a crawl-rate setting in Yandex Webmaster (Yandex’s Crawl-delay page). If Google is crawling too hard, slow responses or errors make it back off.
Should I block GPTBot?
Block it if you do not want OpenAI to use your pages for training. It will not take you out of ChatGPT search: that uses a separate crawler, OAI-SearchBot, and pages a person asks ChatGPT about are fetched by ChatGPT-User (OpenAI’s crawler documentation). The “Allow AI search, block training” preset blocks GPTBot and keeps OAI-SearchBot allowed. To block it by hand, add a group with User-agent: GPTBot and Disallow: /.
Will blocking AI crawlers hurt my search rankings?
Blocking AI training crawlers does not affect Google or Bing, which use Googlebot and Bingbot. Blocking AI search crawlers does remove you from those assistants’ answers, which is a different kind of visibility.
Should I block them?
It depends what you want. A business hoping to be recommended by AI assistants works against itself by blocking search crawlers. A publisher whose archive is the product might reasonably block training crawlers while allowing search, which is what the first preset does.
Where does robots.txt go?
At the root of the domain: https://yourdomain.com/robots.txt. Each subdomain needs its own; a file on the main domain does not cover shop.example.com.
Is anything I build sent to a server?
No. The file is built in your browser. The only server request is the optional “Fetch from a site” import, which reads a public robots.txt and discards it.