llms.txt: What It Is, Who Reads It and Whether You Need One
llms.txt is a proposed Markdown file that sits at yoursite.com/llms.txt and lists the pages an AI tool should read first. It doesn’t control crawling, Google Search ignores it, and server-log studies show most AI bots rarely fetch it. If your CMS can generate one in a click, it costs nothing to have. If you’re hoping it will get your brand cited in AI answers, the studies so far found no link between having the file and being cited.
Clients often ask me about llms.txt, usually because an SEO tool flagged it as “missing.”
What llms.txt is
Jeremy Howard of Answer.AI proposed the format on llmstxt.org in a page dated 3 September 2024. The idea: give language models a short, curated Markdown overview of a site instead of making them dig through navigation, scripts and ads.
The format is simple. The only required part is an H1 with the name of the site or project. After that the spec recommends a blockquote summary, optional notes, and H2 sections containing lists of links, each written as [name](url) with an optional description. A section called “Optional” holds links an agent can skip.
A revised version of the spec went up in August 2026. It added standard link relations so a page can point to its own Markdown version, made it explicit that a file can cover a subfolder, and dropped the special handling of the Optional section. It also states that llms.txt is “used most heavily for software documentation, where coding agents follow them to find API references and tutorials.”
# Example Store
> Example Store sells handmade wool rugs online and ships to the US
> and Canada. This file lists the pages an AI assistant should read
> first.
Customer support: support@example.com
## Shop
- [All rugs](https://example.com/collections/all-rugs): every rug in stock
- [Washable rugs](https://example.com/collections/washable-rugs): machine-washable range
- [Size guide](https://example.com/pages/size-guide): how to pick a rug size
## Policies
- [Shipping](https://example.com/policies/shipping-policy): delivery times and costs
- [Returns](https://example.com/policies/refund-policy): 30-day returns
## Optional
- [Blog](https://example.com/blogs/news): care guides and styling tipsAn example llms.txt for an online store: one H1, a summary blockquote, then H2 sections of links.
One common misunderstanding: llms-full.txt is not part of the spec. Neither version of llmstxt.org mentions it. It is a convention from Mintlify, which says it developed the file with Anthropic to feed full documentation into models.
llms.txt vs robots.txt vs sitemap.xml
| robots.txt | sitemap.xml | llms.txt | |
|---|---|---|---|
| Purpose | Rules on what bots may crawl | List of URLs to crawl | Curated reading list for AI tools |
| Format | Plain-text directives | XML | Markdown |
| Blocks anything? | Yes, for bots that obey it | No | No |
| Used by Google Search | Yes | Yes | No |
Ahrefs summed it up well in its June 2026 study: llms.txt “is not a robots.txt-style directive: it controls nothing and blocks nothing.” If you want to allow or block AI crawlers, that still happens in robots.txt. OpenAI, Anthropic and Perplexity all point site owners to robots.txt in their crawler documentation.
Does Google use llms.txt?
No. Google’s AI optimization guide, published 15 May 2026 and updated 10 July, says you don’t need “AI text files” to appear in Google Search. On keeping one for other services, it says doing so “will neither harm nor help your site’s visibility or rankings in Google Search, as Google Search ignores them.” The same applies to AI Overviews and AI Mode, which Google’s separate AI features page says need no special files either.
John Mueller had been saying this for over a year. In April 2025 he compared llms.txt to the keywords meta tag on Reddit. In June 2025 he posted on Bluesky: “FWIW no AI system currently uses llms.txt.” When people pointed out that some Google developer sites publish an llms.txt file, he replied in January 2026 that it was not an endorsement.
Two things cause confusion here. Google’s Gemini API documentation does host llms.txt files, mainly for coding tools. And Chrome’s Lighthouse added an experimental llms.txt check in April 2026. That check marks a missing file as “Not Applicable.” A third-party tool that fails you for not having one is applying its own rule.
Who actually fetches llms.txt
The best data comes from server logs. Ahrefs looked at 137,210 domains in its Web Analytics product in May 2026. Of the roughly 38,000 sites with a valid llms.txt file, 97% received zero requests for it that month. AI retrieval bots, the kind that fetch pages to answer a user’s question, made up 1.1% of the requests. GPTBot (OpenAI’s training crawler) and the Claude-Code agent were the most frequent AI fetchers. Neither is an AI search bot.
A smaller log study by EZY Research tracked 83 sites for 12 weeks. Over that period OpenAI’s bots requested robots.txt 3,990 times and llms.txt 7 times. Meta’s agent was the one exception, fetching llms.txt slightly more often than robots.txt. EZY sells an llms.txt tool, so a finding that the file rarely gets read cuts against its own interest.
And on the question most marketers care about, SE Ranking analysed about 300,000 domains in November 2025 and found no correlation between having an llms.txt file and how often a site was cited by AI. Removing the llms.txt variable actually made its model more accurate.
Adoption figures vary widely depending on who counted. Ahrefs found 28% among its own customers and called that an upper bound. SE Ranking found 10.13%, and scans of the top 1,000 to 100,000 sites published in September 2026 found between 8% and 10%.
When llms.txt is worth having
- You publish developer documentation or an API. This is where coding agents actually use the file, and where the spec itself says it is used most.
- Your platform generates it for you. Then it costs nothing, and you can keep the content accurate with a few minutes of editing.
- You want a curated list for AI agents that browse on a user’s behalf. This use is still early, but a short, accurate file does no harm.
For a typical Shopify store or B2B manufacturer site, it takes about five minutes, or none on Shopify, where the file already exists. I don’t put it on a priority list above crawl problems, product page content or getting covered on the review sites AI tools cite.
How to add llms.txt on Shopify and WordPress
Shopify. Shopify’s developer changelog (28 May 2026) says every store serves a default file at /agents.md, and /llms.txt and /llms-full.txt point to the same content. To customise all three, edit an agents.md.liquid template; Shopify’s docs advise against adding a separate llms.txt.liquid in most cases. The file is served only on the bare primary domain, not in Shopify Markets subfolders.
WordPress with Yoast. Yoast added automatic llms.txt generation in version 25.3 (June 2025), free for all users. It sits under Yoast SEO, Settings, Site features, AI tools. It isn’t available on multisite installs.
WordPress with Rank Math. The free plugin includes an “LLMS Txt” module. Switch it on from the Rank Math dashboard, then edit the file under General Settings.
WordPress with AIOSEO. The llms.txt generator is in both Lite and Pro. llms-full.txt and the Markdown settings need Pro.
Wix. Wix generates and maintains the file automatically. If you edit it, Wix stops updating it for you.
Whichever route you take, open yoursite.com/llms.txt afterwards. Some sites return their HTML 404 page, or even a copy of robots.txt, at that address.
Should you noindex it?
Mueller suggested it in July 2025: “using noindex for it could make sense, as sites might link to it and it could otherwise become indexed, which would be weird for users.” Google may crawl and index the file like any other text file. Sending an X-Robots-Tag: noindex header keeps it out of search results without stopping AI tools from reading it.
What to do instead if you want AI visibility
Google’s guide says that, from Google Search’s perspective, optimizing for generative AI search is “still SEO.” For other AI assistants, the work that moves results is the same groundwork: make sure their crawlers can reach you in robots.txt and at your CDN, put product facts in plain HTML, and earn coverage on the sites those assistants cite. When I sampled eight buying questions on ChatGPT in October 2026, 31 of the 64 citations went to brand websites and 18 to niche review sites.
Our GEO services start with that kind of sampling for your category.
For the bigger picture on how SEO and AI search overlap, see our piece on SEO vs GEO vs AEO.
Quick answers
Is llms.txt mandatory? No. No search engine or AI company requires it.
Is llms.txt still used? Mostly on developer documentation sites, where coding agents read it. In Ahrefs’ May 2026 logs, 97% of files got no requests at all.
Is it llm.txt or llms.txt? llms.txt. The spec and every platform above use /llms.txt.
Does llms.txt really work? Not for rankings or AI citations, based on the studies above. It can help an agent find the right documentation page faster.
