llms.txt
llms.txt is a proposed standard: a markdown file placed at the root of a website that gives large language models a curated summary of the site and links to its most important content. It was proposed by Jeremy Howard of Answer.AI in September 2024 as a way to help AI systems use a site's key pages without wading through navigation, scripts and boilerplate HTML.
The format is deliberately simple. The file lives at yourdomain.com/llms.txt and contains a title, a short summary in a blockquote, and lists of links to key pages with one-line descriptions of each. A companion convention, llms-full.txt, goes further and includes the full text of the site's important content in a single file. The logic is that model context windows are finite and HTML is noisy, so a curated, machine-readable map lets an AI system spend its context on your actual content.
The honest status report: adoption by site owners is real, confirmation by AI engines is not. At the time of writing, no major AI provider, including OpenAI, Google and Anthropic, has publicly confirmed that its assistants read llms.txt when generating answers. Some AI crawlers have been observed fetching the file, but a fetch is not evidence that the contents influence what an assistant says. Google's John Mueller has publicly compared it to the keywords meta tag, which is about as blunt as scepticism gets.
It is also worth separating llms.txt from robots.txt, because the two are often conflated. robots.txt controls access: it tells crawlers what they may and may not fetch. llms.txt grants no permissions and blocks nothing; it is a suggestion about what is worth reading. Adding one does not protect your content from AI training, and omitting one does not hide anything.
Should you add one? The cost is an hour of work and the risk is essentially zero, so it is reasonable cheap insurance, particularly for documentation-heavy sites. Just be clear-eyed about the return: there is currently no public evidence that llms.txt alone improves how AI assistants describe or cite you, and anyone selling it as a visibility fix is ahead of the facts.