About LLMs Text Generator

What is this site?

Tools and guides covering llms.txt, llms-full.txt, robots.txt and XML sitemaps, plus a sourced news archive. The tools are free, with no account. Optional paid AI features are planned, and the free tools stay free.

• llms.txt generator: builds a curated index from a website, a sitemap or your own list of URLs.
• llms-full.txt generator: scans a site, lets you choose sections, and builds the full-text companion file in Markdown.
• Validator: checks structure, links and serving, with a published scoring method.
• robots.txt generator and tester: crawl rules for search engines and the current AI crawlers.

How do the tools work?

When you start a generator, the server reads your sitemap (or your homepage links, or the list you paste) and then fetches each page in turn, one at a time. From each page it takes the title, the meta description and, for llms-full.txt, the main content, which it converts to Markdown. Navigation, headers, footers, forms and scripts are removed. Progress streams back to your browser as it goes, and pressing Cancel stops the crawl on the server as well.

The crawler identifies itself as LLMsTextGeneratorBot with a link to this page, and it follows each site’s robots.txt according to RFC 9309: a disallowed page is skipped and reported, a missing robots.txt means no rules, and a robots.txt that answers with a server error means the site is treated as off limits. Crawl-delay is honoured up to ten seconds.

There are limits to keep it a polite visitor: 12 seconds per page, 3 MB per page, free runs of up to 50 pages and 3 runs a day per visitor (paid plans, when they open, read up to 250 pages a run), one job at a time per visitor, and a cap on requests per minute. The tools only fetch public web addresses on standard ports; anything resolving to a private network is refused.

What is stored?

No copy of the pages you point the tools at, and no copy of your file. Pages are held in memory while your file is built, the result is sent to your browser, and both are discarded when the request ends. What the server does keep is operational: short-lived pseudonymous counters for the free limits, daily usage totals and error logs. The robots.txt generator and tester do their work in your browser. The last few runs of each tool are remembered in your own browser so you can repeat them; you can clear that list at any time. Details are on the privacy page.

How are the guides written and kept current?

The guides are based on primary sources: the llms.txt proposal at llmstxt.org and the V2 change notes, RFC 9309 for robots.txt, and each AI company’s own crawler documentation. Every guide shows the date it was last updated, and the crawler list in the robots.txt tools records the date its entries were checked. When the proposal changes, the guides and the validator change with it: the August 2026 V2 revision is reflected throughout.

There is a lot of overstatement in this area: ranking claims, invented statistics, urgency about a convention that is two years old and unevenly adopted. The guides avoid that. Where something is unproven, they say so, and where a file will not help you, they say that too. The SEO guide is the clearest example.

What does this site not do?

• It does not promise rankings, traffic or citations. Nobody can.
• It does not add its own name or links to the files you generate. They are yours.
• It does not track you, show ads or load anything from another domain.

Questions, corrections or a crawler problem with your site: write to enquiry@llmstextgenerator.com.