Pull up your server logs from the last month and search for requests to `/llms.txt`. On a typical business site you’ll find a handful. A few will be SEO tools checking whether the file exists. Maybe a competitor got curious. Very few will come from the AI crawlers the file was supposedly written for.
That gap, between how much the file gets talked about and how little it gets read, is the most useful place to start. llms.txt isn’t useless. It was sold to the wrong audience for the wrong job.
Where the file came from
Jeremy Howard, co-founder of Answer.AI, proposed llms.txt in September 2024. The problem he had in mind was narrow and real. When a language model works with a website on the fly, for example a coding assistant told to use a library’s documentation, it burns a lot of its limited context on menus, cookie banners, scripts and layout markup. A plain markdown file at the root of the domain could give it a curated map instead.
The proposal at [llmstxt.org](https://llmstxt.org) is short. The only required piece is an H1 with the name of the site or project. After that it suggests a blockquote with a brief summary, some optional free text, and H2 sections holding lists of markdown links, each with an optional note after a colon. A section titled “Optional” marks links a model can drop when it’s tight on space. The proposal also suggests offering clean markdown copies of key pages, usually at the same URL with `.md` on the end.
That’s all of it. No directives, no permissions, nothing a crawler is obliged to follow.
How a docs convenience became a GEO tactic
During 2025 the idea left the developer docs world and showed up in GEO playbooks. The pitch was easy to repeat: robots.txt is for crawl control, sitemap.xml is for indexing, and llms.txt is the file that gets you cited by ChatGPT and Perplexity. SEO plugins added one-click generation. Audit tools started flagging a missing file as a problem.
The numbers never supported that story. SE Ranking checked 300,000 domains and found the file on about 10% of them, and among the 50 domains cited most often in AI answers, exactly one had it. Otterly added an llms.txt to a test domain and tracked AI bot traffic for 90 days: 84 of 62,100 bot visits touched the file, around a tenth of a percent. Otterly later took llms.txt off its own GEO audit checklist. [Limy](https://limy.ai/blog/llms-txt-in-2026-the-full-guide) went bigger, reviewing more than 500 million AI bot events over a 90-day window, and counted 408 requests for the file. GPTBot, ClaudeBot and PerplexityBot mostly skipped it and went straight for HTML.
One report points the other way. Profound has said crawlers from Microsoft and OpenAI do fetch llms.txt and llms-full.txt. Fair enough, but fetching a file and using it to decide who gets cited are two separate claims, and only the first has any evidence behind it.
What Google put in writing
Google closed its side of the debate this year. On May 15, 2026, Search Central published a guide to [optimizing for generative AI features in Google Search](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide), including a list of tactics site owners can ignore. llms.txt is on that list. The guide says you don’t need new machine-readable files, AI text files or markdown versions of pages to appear in Search, AI Overviews and AI Mode included, because Search doesn’t use them.
A revision in June adjusted the tone. Google now says it’s fine to keep an llms.txt for other services that use one, and that it won’t help or hurt your Google visibility because Search ignores it.
None of this surprised anyone who’d been following Google’s staff. Back in 2025 John Mueller compared the file to the keywords meta tag, which in SEO circles is close to the harshest comparison available.
For Google, then, the effect is zero in both directions. Treat it that way.
The reader it was actually built for
Google’s guidance is about Google Search. It says nothing about the tools that read the web on someone’s behalf at the moment a question gets asked, and that category is growing fast.
Picture a developer pasting your docs URL into a coding assistant and asking it to build an integration. Or someone telling an agent to compare three vendors and fill in a spreadsheet. Or a company building a support bot on a RAG pipeline that needs to pull in a help center. In each case a program is fetching pages live, with a finite context window and no appetite for a heavy HTML page where the real answer is four paragraphs long.
This is where llms.txt earns its keep. It helps once a visitor is already on the way. It has no influence over who decides to visit.
That’s also why the files getting genuine use sit mostly on developer documentation. API providers, SDK maintainers and the AI labs themselves publish them, and some docs platforms generate them out of the box. Those audiences arrive with agents already attached.
Which sites it’s worth an afternoon for
If you publish technical documentation, run an API or SDK, or sell a product people integrate with, ship one, along with markdown copies of your docs pages. Your users are already pointing assistants at your content. A good file shortens the trip from their question to working code.
If you run a B2B service firm, a SaaS marketing site or a content-heavy publication, it belongs near the bottom of the list. Don’t expect more citations. It may help agents that prospects send to research you, and the writing process has a side benefit covered below. Give it an afternoon, not a sprint.
If you run a local business or a straightforward online store, you can skip it and lose nothing you could measure. Crawlable pages, accurate product data, reviews and pages that answer real buyer questions will all do more.
Whichever group you’re in, if the justification on the table is “this will get us into ChatGPT answers,” stop the conversation there. Nothing backs that up.
Writing a file that’s worth serving
Most llms.txt files online come from a plugin that pours the sitemap into markdown. You get a stack of bare URLs, which tells a model nothing sitemap.xml didn’t already. If you’re going to publish one, write it yourself. Here’s a sketch for a made-up payroll API company:
“`markdown
# Kestrel Payroll
> Kestrel is a payroll API for software platforms that pay contractors in the EU and UK. It handles tax withholding, local compliance filings and payouts in 14 currencies.
Kestrel does not offer a standalone payroll app for small businesses. Everything runs through the API or the partner dashboard.
## Docs
– [Quickstart](https://kestrelpayroll.example/docs/quickstart.md): create a sandbox account and send a first test payout in about 15 minutes
– [Authentication](https://kestrelpayroll.example/docs/auth.md): API keys, OAuth for partner platforms, key rotation
– [Payout lifecycle](https://kestrelpayroll.example/docs/payouts.md): statuses, failure reasons, retry rules
## Company
– [Pricing](https://kestrelpayroll.example/pricing.md): per-payout fees by country and volume tier
– [Security and compliance](https://kestrelpayroll.example/security.md): SOC 2 report access, data residency options
## Optional
– [Changelog](https://kestrelpayroll.example/changelog.md)
– [Brand assets](https://kestrelpayroll.example/press.md)
“`
The summary covers what the company does, who it serves and where its limits are. That line about not selling a small business app does real work: it keeps an agent from confidently telling a shop owner something false. Getting those two sentences agreed internally is often harder than it sounds. If your team spends an hour arguing over them, you’ve found a positioning problem that matters far beyond this file.
Each note after a link says what the page answers. “Pricing” on its own is a label. “Per-payout fees by country and volume tier” is information a model can act on.
Every URL is absolute. The file regularly gets fetched and read with no surrounding context, so relative paths break.
It stays short. Choose the 20 to 60 pages that someone evaluating or using your product would actually open. Beyond that, the curation that makes the file worth reading starts to disappear.
The markdown copies deserve a word of caution. They’re where agents get most of the practical benefit, but they also create a second version of every page, and Google’s guide notes that it can discover and index files beyond HTML. To keep the copies from competing with your real pages, serve them with a `Link` header using `rel=”canonical”` that points to the HTML version, or with an `X-Robots-Tag: noindex` header. Agents can still fetch them, and they stay out of search results.
Last, hand the file to whatever process already maintains your sitemap and redirects. Links in llms.txt that lead to 404s send an agent into dead ends, which is a poor first impression to leave with anything that’s about to summarize your company.
Checking whether anything reads it
You don’t have to trust vendor studies, or this post. Your logs will answer the question.
Filter requests for `/llms.txt`, `/llms-full.txt` and your `.md` paths over 60 to 90 days, and group them by user agent. Separate known AI crawlers from generic fetchers and SEO tools. For a cleaner signal, include one link in the file to a URL that appears nowhere else on the site. Anything that requests it got there by reading the file.
If three months pass and nothing reads it, you’ve spent an afternoon. If agents do turn up, you’ll see which pages they go after, and that tells you more about your AI audience than most dashboards currently sold under the label of AI visibility.
