llms.txt is a plain-text markdown file that lives at the root of your domain and gives AI systems a short, curated map of your site: what it is, which pages matter, and what each one covers. Ours is 41 lines. It took 25 minutes to ship and it regenerates on every deploy.
The proposal is young and adoption is uneven, which is why the advice around it is so muddy. Half the posts treat it as a ranking hack. The other half dismiss it as pointless. Both are wrong in ways that cost you something.
Here is what the file actually is, exactly what to put in it, and the structure we generate for avakata.agency.
What is llms.txt, exactly?
llms.txt is a markdown file served at yourdomain.com/llms.txt, proposed by Jeremy Howard in September 2024. It contains a single H1 with your site name, a short blockquote summary, and a few H2 sections of links with one-line descriptions. The point is to give a language model a context-window-sized orientation to your site in one request, instead of forcing it to crawl twenty pages and guess.
The audience is inference-time readers as much as training crawlers — assistants that fetch a page on a user's behalf, right now, mid-conversation. A model that reads 41 clean lines knows more about our agency than one that parses our homepage markup and navigation soup.
There is a companion convention, llms-full.txt, which inlines full page content instead of linking to it. We skip it. Our pages already render clean, readable HTML, and maintaining a second full-text mirror of the site is a job with no owner.
One framing that helps: it is a README for your website. Write it the way you would write for a competent stranger who gave you 60 seconds.
What llms.txt is not
llms.txt is not a permissions file and not an index. robots.txt tells crawlers what they may fetch. sitemap.xml enumerates every URL you have. llms.txt curates the 10 to 15 pages you actually want a model to read, with enough description attached to use them correctly. It grants nothing, blocks nothing, and ranks nothing on its own.
That last clause deserves its own sentence: there is no credible evidence that publishing llms.txt improves rankings or citation rates by itself. Anyone selling it as a growth lever is selling you 25 minutes of work at strategy prices.
It also does not override anything. If your robots.txt blocks GPTBot, an inviting llms.txt does not sneak you back in. The two files answer different questions — may you read, and what should you read first.
Treat it as documentation with a small expected payoff and a near-zero cost, in the same family as alt text and a well-written 404 page.
Use the structure the spec expects
The format is strict on shape and loose on length. One H1 with the site name. One blockquote holding a one-or-two-sentence summary. An optional short paragraph of context. Then H2-titled sections, each a markdown list of links in the form title, URL, colon, one-line description. Keep the whole file under roughly 2,000 tokens so it fits comfortably inside a working context alongside the user's actual question.
Order sections by what you want read first. Ours runs Services, then Field notes, then About. A model that stops reading halfway through still got the pitch.
The spec reserves an H2 called Optional for links that can be skipped when context is tight. Use it. It is the only priority signal the format gives you.
Two mistakes we see weekly: serving HTML instead of plain markdown, which makes parsers give up, and opening with a marketing paragraph instead of the summary. Plain text, top-loaded, no exceptions.
What to put in it, line by line
Our file is 41 lines: one H1, one blockquote positioning sentence, a three-sentence context paragraph, four service links, eight selected posts, two about links, and one Optional entry. The selection rule is simple. A page earns a line only if we would be happy seeing an AI quote it word for word to a prospect.
Descriptions do the real work. Write each one as an answer, not a slogan. "Flat-fee GEO retainer at $1,500 a month, cancel monthly" gives a model something to say. "Solutions that drive growth" gives it nothing.
We cap descriptions at 160 characters, the same discipline as a meta description. If a page cannot be described in one honest line, that is a page problem, not a file problem.
Eight posts, not all 67. We pick the ones with the highest citation counts in our monthly audits and swap them as the data moves.
What to leave out
Leave out anything you would not want summarized: tag archives, pagination, legal boilerplate, thin location pages, gated content, and the long tail of posts that exist for one narrow query. A 40-line file gives every line weight. A 400-line file is a sitemap with extra steps, and dilution is the main way people break this format.
Leave out keyword stuffing too. The reader is a language model. It notices, and the file reads worse to every human who checks it.
One useful test: paste your file into a chat model and ask it to describe your business and pick the three pages a prospect should read first. If the answer embarrasses you, the file is not done.
When in doubt, cut. You can add the line back next deploy.
Does anything actually read it?
Honest answer: adoption is real but thin. In our last 30 days of logs, llms.txt was fetched 63 times by AI-associated user agents — ClaudeBot, GPTBot, PerplexityBot, and a handful of retrieval tools — against roughly 41,000 total bot requests sitewide. That is not a traffic channel. It is a small, cheap bet that pays off if the convention hardens.
Several large AI companies publish their own llms.txt files, which tells you the people building the readers believe the format has a future. It does not tell you the readers weight it heavily today.
Our position: ship it because it is nearly free, not because it is a strategy. The strategy is still answer-first structure, schema, and freshness on the pages themselves.
If you want a benchmark, watch your own logs for a month before and after shipping. The fetch count is the entire funnel report.
Generate it. Do not hand-write it
Treat llms.txt as a build artifact, not a document. Our Astro build queries the CMS at deploy time, takes the current service pages, pulls the eight most-cited posts from the audit table, truncates every description at 160 characters, and writes the file. It is about 40 lines of build code, written once in an afternoon.
Hand-written files rot. We watched a client's hand-made llms.txt drift out of date in six weeks — two dead links and a retired service, still advertised to every model that asked. A stale map is worse than no map, because the reader trusts it.
The whole job, then: write the generator, wire it into deploy, and fold the output into your monthly audit. After that it costs 25 minutes of attention a quarter.
The file is the easy part. Knowing which eight posts deserve to be in it is the actual work, and that is the half we run continuously for every Avakata retainer client.