llms.txt explained: what it is, an example, and whether you need it

llms.txt is a plain Markdown file placed at the root of a website (/llms.txt) that gives AI tools a short, curated map of the site: what it is, and which pages matter. The idea was proposed by Jeremy Howard on September 3, 2024, and the proposal at llmstxt.org has been revised since^[1].

The reasoning behind it is practical. Web pages are full of navigation, scripts and layout that a language model has to wade through, and a model's context window can't hold a whole site. A hand-picked list of links with one-line notes is easier to use. The proposal also suggests offering clean Markdown copies of important pages at the same URL with .md added^[1].

It helps to be clear about what it is not. llms.txt is not a standard like robots.txt, it does not grant or block access to anything, and no search engine has said it uses it to rank or cite pages. If you want to control which AI bots can read your site, that is robots.txt; see AI crawlers and robots.txt.

An llms.txt file has up to four parts, in a fixed order. Only the first is required^[1].

The proposal allows the file at the root or in a subpath such as /docs/llms.txt^[1]. Use absolute URLs in links, keep each note to one line, and link to pages that answer real questions rather than to every URL in your sitemap.

Here is a complete example for a fictional accounting software company that sells in several countries. Notice what it leaves out: blog archives, tag pages and legal boilerplate.

# Example Ledger

> Example Ledger is cloud accounting software for small businesses and
> accountants. It handles invoicing, bank reconciliation, payroll and sales
> tax / VAT / GST returns, and is sold in the US, Canada, Australia and Brazil.

- Pricing is per company, billed monthly or yearly.
- Payroll is available in the US and Australia only.

## Product

- [Features overview](https://www.example.com/features): What the product does, by module
- [Pricing](https://www.example.com/pricing): Plans and what each includes
- [Integrations](https://www.example.com/integrations): Banks, payment apps and e-commerce platforms

## Help center

- [Getting started](https://www.example.com/help/getting-started.md): Set up a company in 15 minutes
- [Bank feeds by country](https://www.example.com/help/bank-feeds.md): Supported banks per market
- [Migrating from spreadsheets](https://www.example.com/help/migrate.md): Import steps and templates

## Optional

- [Company history](https://www.example.com/about): Founders and timeline
- [Changelog](https://www.example.com/changelog): Release notes

The free llms.txt generator drafts a starting file from your homepage, using your own title, description and navigation links, and its validator mode checks an existing file against the format. You then trim and annotate it.

No. Google says you don't need machine-readable files, AI text files or Markdown to appear in Google Search, including its generative AI features, and that creating them will neither harm nor help your visibility because Google Search ignores them^[2]. Google added that clarification to its AI optimization guide on June 15, 2026^[3].

That covers AI Overviews and AI Mode, which run on Google Search. It is a statement about Search, not about every Google product. Chrome's Lighthouse tool, for example, added an llms.txt check to its experimental Agentic Browsing audits. That audit marks a missing file as not applicable, since the file is optional, and only flags a server error when fetching it^[4]. In other words, Google's tooling treats llms.txt as optional plumbing for agents, not as a search signal.

Mostly people and tools that are pointed at it on purpose, rather than crawlers that look for it automatically. The best public evidence comes from server logs. Ahrefs studied about 137,000 domains using its analytics product and found that 28% published an llms.txt file, but 97% of those valid files received zero requests in May 2026^[5]. Ahrefs notes its sample skews toward AI-aware sites, so adoption across the wider web is likely lower^[5].

Where llms.txt does get used is in hands-on work: a developer pastes a docs site's llms.txt into a chat assistant or coding tool to give it context, or an agent framework fetches it while exploring a site. Several AI and infrastructure companies, including Anthropic and Cloudflare, publish one for their own documentation^[5]. That use is real, but it is a different job from being recommended in an AI answer about your category.

What does drive AI recommendations is better understood: being crawlable by the AI search bots, being described consistently on the sites engines cite, and having clear pages that answer the questions buyers ask. The guide to generative engine optimization covers those levers.

Create one if it is cheap for you and your site has content worth summarizing; skip it, or leave it for later, if you'd be building it from scratch for a five-page site. The cost is small, the measurable benefit for AI visibility is close to zero today, and the downside is mainly upkeep.

• Cost: usually under an hour with a generator, plus a few minutes whenever key pages change.
• Risk: almost none for Search, per Google^[2]. Ahrefs points out one trade-off: a tidy map of your best content is also easier for competitors to scrape^[5].
• Upside: better context when people or agents deliberately use your file, and a ready summary if more tools adopt the proposal.

Start from your real priorities, not your full sitemap. These steps take most teams under an hour:

1. List the 10 to 30 pages a new customer or partner most needs: product, pricing, key help articles, policies such as shipping or returns, and contact details.
2. Write a two or three sentence summary that says what you do, for whom and where. Name your markets and languages if they matter.
3. Draft the file with the llms.txt generator, or by hand, using the H1, blockquote and H2 link-list structure. Add the pages from step 1 that your navigation doesn't link.
4. Move nice-to-have links under ## Optional.
5. Publish it at https://yourdomain/llms.txt as plain text and check it returns HTTP 200, not a redirect to your homepage or a 403 from a firewall.
6. If you can, publish clean .md versions of the most important pages and link to those.
7. Add a reminder to review the file whenever you change pricing, launch a product or retire pages.

Validation checklist

• Exactly one H1, and it is the first heading.
• No headings inside the free-form notes; sections start with H2.
• Every list item in an H2 section is a Markdown link, with an optional : note.
• Links are absolute, on your own domain where possible, and return 200.
• The file is reachable by AI bots: check robots.txt and edge rules with the AI crawler checker.

Then measure what matters. An llms.txt file won't show up as a change in AI answers, so judge your AI visibility by whether ChatGPT, Google AI Overviews, Google AI Mode, Gemini, Perplexity and Claude mention you for real buyer questions. The free AI visibility checker runs a quick sample, and Glossy Data's methodology explains how paid tracking separates real movement from run-to-run noise.