llms.txt: what it is and whether you need one

The proposal came from Jeremy Howard of Answer.AI in September 2024. His argument: HTML pages are a poor way for a language model to learn about a site. They arrive wrapped in navigation menus, cookie banners, script tags and tracking pixels, and a model working within a limited context window wastes most of its attention on boilerplate. A single clean markdown file at the root of the domain hands over the essentials directly.

The format is deliberately simple. An H1 with the site or company name, then a blockquote containing a one-or-two-sentence summary of what you do. After that come H2 sections grouping your key links, with each link followed by a one-line description. A companion file, llms-full.txt, can carry the full text of those pages for models that want everything in a single request.

It is not robots.txt or sitemap.xml under another name. Those two files are mechanical: one tells crawlers what they may not access, the other lists every URL you have with no opinion about which matter. llms.txt is editorial. It is the shortlist of pages you would hand a well-informed stranger, with context on why each one is there. There is a fuller definition in our glossary entry on llms.txt.

llms.txt is a community proposal, not a web standard. No standards body has adopted it, and no major AI engine, including OpenAI, Anthropic, Google or Perplexity, has confirmed that it uses the file for retrieval or for training.

The supply side looks healthy: thousands of sites publish the file, helped along by documentation platforms that generate it automatically. The demand side does not. Server log analyses consistently show AI crawlers fetching llms.txt rarely, if at all, and Google's John Mueller has publicly compared the idea to the keywords meta tag, a signal search engines learned to ignore because site owners cannot be trusted to describe themselves neutrally.

So when a vendor tells you llms.txt boosts your AI visibility or your rankings, ask for evidence. We have not seen any that survives scrutiny. Correlation studies circulate, but sites that publish llms.txt files are also sites that care about structured, current content, and that second trait explains any visibility gains far more plausibly than the file does.

Given all that, the honest recommendation is still yes, publish one. The reasoning is about cost, not proof. A first draft takes about 20 minutes for a typical marketing site, and the result is a small static asset. No engine penalises you for having one.

The stronger argument has nothing to do with AI engines. Writing an llms.txt forces a question most sites never answer explicitly: if a stranger could read only ten of your pages, which ten? Teams that sit down to write the file tend to discover that their pricing page is vague or that their about page was last touched three years ago. The file takes 20 minutes; the audit it triggers is the real value.

There is also the accuracy problem the file gestures at. In our July 2026 test of 30 brands, AI described 27 of the 30 (90 percent) with at least one materially false claim when answering from memory, and 13 of 30 (43 percent) even with live web search running. Clear and current pages are what fix that, and llms.txt nudges you towards maintaining them. If an engine does start honouring the file, early adopters inherit the position at no extra cost.

Start with the skeleton the proposal specifies. Line one is an H1 with your company or site name. Directly beneath it goes a blockquote of one or two sentences stating what you do and for whom, in plain factual language. That blockquote is the single highest-leverage part of the file, so write it the way you would want a model to repeat it.

Then add H2 sections grouping your key links. Sensible groupings for a commercial site: products or services, pricing, documentation or guides, and company information. Under each heading, list links in markdown format with a one-line description after each. Write descriptions as verifiable statements, not slogans. 'Growth plan pricing and what each tier includes' beats 'Discover our flexible plans'.

Most sites need between ten and thirty links. The proposal allows an H2 called Optional for secondary material, which is a useful home for anything a model can skip when context is tight. Serve the file at the root of your domain as plain text or markdown, and keep it wherever your site content lives, so it gets updated when pages change rather than rotting in a forgotten deploy.

The most common failure is dumping every URL into the file. That reproduces your sitemap and throws away the one thing llms.txt offers, which is curation. A model handed 4,000 undifferentiated links learns nothing about what matters; the file only works as a shortlist.

Bare links without descriptions are nearly as bad, because the description supplies the context a URL cannot. Ad copy in the descriptions is a subtler version of the same mistake. A model grounding its answers in your file can do nothing useful with 'the leading platform for modern teams', but it can repeat 'B2B payroll software for UK companies with 10 to 200 employees'.

Two more worth flagging: letting the file go stale, which turns a trust signal into a liability the moment a model quotes your 2024 pricing, and treating the file as a rankings play, which sets expectations no current evidence supports. Publish it and keep it current, but expect nothing from it directly.

Keep proportion. llms.txt is a minor tactic within generative engine optimisation, the wider discipline of making sure AI engines describe and recommend you accurately. The levers with real evidence behind them are less glamorous: accurate, up-to-date facts on your own pages, and third-party sources that corroborate them.

The place to start is measurement, not file creation. Run your brand through our free AI visibility checker, which needs no login and shows you what Claude, using live web search, says about your brand right now (checks on other engines are rolling out). Fix what it gets wrong, then check again. An llms.txt file is a sensible 20 minutes along the way; it is not the work itself.