# Common Crawl looks inside 584,107 llms.txt files

> Common Crawl read every llms.txt in its July 2026 crawl: most come from plugins or templates, many have no links, and some try to act like robots.txt.

Source: https://llmstextgenerator.com/llms-txt-news/common-crawl-llms-txt-analysis/
Last updated: 17 October 2026

Common Crawl analysed every non-empty llms.txt in its July 2026 crawl. Most were made by plugins or templates, nearly a quarter had no links, and some tried to give crawlers orders.

Happened on 31 August 2026

## What happened?

Common Crawl, the non-profit that publishes an open archive of the web, looked at every llms.txt it fetched in its July 2026 crawl (CC-MAIN-2026-30): 584,107 non-empty files out of 598,298 text responses. The analysis was written by Malte Ostendorff, a senior research engineer there, and published on 31 August 2026.

## What did the files contain?

| Finding | Share of files |
| --- | --- |
| Produced by a plugin or site template | 68.27% (Wix alone: 41.34%) |
| No links at all | 22.56% |
| A `## Optional` section | 9.71% |
| Rules the proposal does not define (rate limits, copyright notices, citation demands) | 6.59% |
| Naming a specific crawler | 1,570 files (0.27%) |

664 files were written in robots.txt syntax under the wrong filename. GPTBot was named in 867 files, and one in five of those told it to stay away. Common Crawl then checked the robots.txt of the 32 sites whose llms.txt files told its own crawler, CCBot, to keep out: on 17 August 2026, none of them blocked CCBot in robots.txt, where the rule would actually apply.

The analysis also screened for prompt injection, text written to manipulate an AI reader. Ten files triggered the top severity level, and four of those were genuine attempts.

## What does it mean for your site?

**llms.txt cannot block or permit anything.** In Common Crawl’s words, “the naming invites the mistake: `llms.txt` is not an access-control mechanism.” If you want to keep a crawler out, the rule belongs in [robots.txt](https://llmstextgenerator.com/robots-txt-generator/).

Keep llms.txt to what it is for: a short, curated list of your useful pages with a sentence on each. A file with no links, which nearly a quarter of these were, gives a reader nothing to follow. If a plugin wrote yours, open it and check what it actually says; the [validator](https://llmstextgenerator.com/llms-txt-validator/) will flag missing links and structure problems.

## Sources

**Source:** [Common Crawl: a content analysis of llms.txt files from the July 2026 crawl archive](https://commoncrawl.org/blog/a-content-analysis-of-llms-txt-files-from-the-july-2026-crawl-archive)

**Coverage:** [Search Engine Journal, 8 September 2026](https://www.searchenginejournal.com/common-crawl-llms-txt-robots-txt/588786/)

[All llms.txt news](https://llmstextgenerator.com/llms-txt-news/)