--- title: "llms.txt After Google's Guidance" description: "llms.txt guide for 2026: what Google says, who actually reads the file, a version 2 example, and best practices for publishing and measuring it." canonical: "https://zerply.ai/resources/blog/llms-txt-guide" author: "Anshul Motwani" date: "2026-10-16T09:14:50+00:00" image: "https://storage.zerply.ai/teams/92/blogs/400/fb3cfeeeef308390-1791278624823-zerply-blog-banner-llms-txt-guide-2026-10-06_2x.png" --- # llms.txt After Google's Guidance > **Direct answer:** llms.txt is a Markdown file that gives AI agents a short, curated map of a website. Google Search ignores it, and Google says it neither helps nor harms rankings in AI Overviews or AI Mode. The file earns its place when agents, especially coding agents, read your docs. Publish it for that audience, link agents to it, and check your logs before investing more. In May 2026, Google gave site owners two messages about the same file. Google Search published a guide to generative AI features that lists llms.txt under things you can ignore. In the same month, Chrome's Lighthouse added an llms.txt check to its agentic browsing audits. Both are true, because they address different readers. Google Search ranks pages from its own index and has no use for the file. AI agents working through a site on someone's behalf are the audience the proposal was written for. This llms.txt guide separates those two audiences, shows who actually fetches the file today, and walks through writing, linking, and measuring one. ## What is llms.txt? An **llms.txt file** is a Markdown file, usually at `/llms.txt`, that summarizes a site and links to clean, text-only versions of its most useful pages. It is written for **AI agents**, software that reads and acts on websites for a person, such as a coding assistant pulling API documentation into its answer. Jeremy Howard of Answer.AI proposed the format in September 2024. [The llms.txt proposal](https://llmstxt.org/) reached version 2 on August 10, 2026. It is still a proposal, and no standards body has ratified it. Two things it gets confused with: - **A robots.txt rule.** llms.txt controls nothing. It cannot allow or block a crawler. Crawler access still lives in robots.txt and your firewall. - **A sitemap.** A sitemap lists every indexable page for search engines. llms.txt lists a hand-picked set of links small enough to fit in a model's context window. The proposal's own framing is that agents use the file on demand, while helping a user with a task, rather than as a crawling instruction. ## Does llms.txt help SEO? No. llms.txt for SEO is a dead end in Google Search, because Google Search does not read the file. [Google's guide to optimizing for generative AI features](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide) is explicit on four points: - You do not need new machine-readable files, AI text files, or Markdown to appear in Google Search, including AI Overviews and AI Mode. - Creating and maintaining llms.txt for other services is fine. - The file will neither help nor harm your visibility or rankings in Google Search. - Google may still crawl and index the file like any other URL it finds, which does not mean it treats the file specially. When [Lily Ray asked John Mueller](https://www.linkedin.com/posts/lily-ray-44755615_got-some-interesting-answers-from-john-mueller-ugcPost-7462840528913494019-V_Iw/) about the mixed signals from Google, he described llms.txt as "not done for search." He framed it as a stopgap that saves tokens for AI coding tools reading developer documentation. ## What are the common myths about llms.txt? The claims around llms.txt are easier to check now that Google, Chrome, and the AI vendors have documented their positions. This myth-vs-reality table tests each one against those sources. | Myth | Reality | Source | | --------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------- | | llms.txt improves visibility in AI Overviews and AI Mode | Google Search ignores the file. It neither helps nor harms rankings. | Google Search Central | | AI search bots look for llms.txt the way crawlers look for robots.txt | In Ahrefs' May 2026 data, AI bots made zero requests for llms.txt files that did not exist. Retrieval bots such as OAI-SearchBot made 1.1% of the fetches that did happen. | Ahrefs | | Lighthouse penalizes sites without an llms.txt file | A missing file is marked not applicable. Lighthouse flags the audit only when the server errors while fetching it. | Chrome for Developers | | llms.txt can block or allow AI crawlers | The file controls nothing. robots.txt and firewall rules handle access. | llms.txt proposal | | The major AI labs have committed to reading it | OpenAI's and Anthropic's crawler documentation covers robots.txt controls and never mentions llms.txt. Both still publish llms.txt files for their own developer docs. | OpenAI, Anthropic | | Publishing the file is enough | Agents fetch llms.txt when a link, index, or instruction points them to it. An unlinked file is unlikely to be found. | Ahrefs | Chrome's [llms.txt audit documentation](https://developer.chrome.com/docs/lighthouse/agentic-browsing/llms-txt) explains the reasoning behind the check: without the file, agents may spend more time crawling a site to work out its structure. The audit still treats the file as optional for now. ## Who actually reads llms.txt? Coding agents and training crawlers read llms.txt far more often than AI search bots do. That is the clearest pattern in Ahrefs' study of 137,000 domains. In its [llms.txt study](https://ahrefs.com/blog/llmstxt-study/), Ahrefs analyzed server logs and live traffic across 137,210 domains for May 2026. Its findings: - **28% of domains published a valid llms.txt file.** Ahrefs treats this as an upper bound, since its customers skew more technical than the web overall. - **97% of those files received zero requests** in the month. No bots and no humans. - **Of the requests that did arrive, 96% came from bots.** Slackbot, unfurling links shared in chat, fetched llms.txt files more often than PerplexityBot did. - **AI tools made 19.5% of the fetches.** AI agents made 10.5%, training crawlers 5.3%, AI assistants 2.5%, and AI retrieval bots 1.1%. - **GPTBot fetched the most** of any AI bot, and Claude-Code, Anthropic's coding agent, out-fetched every AI search and assistant bot. - **Lighthouse itself accounted for roughly 1 in 1,000 fetches.** Ahrefs flags a limit that is worth repeating. A fetch proves only that the file was downloaded, so every figure is a ceiling on real use. When read together, the data points to one audience. If developers point coding agents at your documentation, the file has readers. If the goal is a citation in ChatGPT search or Perplexity, the retrieval bots behind those answers barely touch it. Effort aimed at citations belongs in the pages themselves, which our guide to [content AI search cites](https://zerply.ai/resources/blog/content-cited-by-ai) covers in detail. ![](https://storage.zerply.ai/teams/92/blogs/400/1533a6fa1be76e6a-1791278665345-image.png) ## When is llms.txt worth publishing? llms.txt is worth publishing when agents need a guided path into content you own, and when you will keep the file current. Publish one if: - You have developer documentation, an API reference, or SDK guides that coding agents read - Your docs platform or CMS generates the file for you, so the cost is close to zero - Customers or partners ask agents to work with your product and need a reliable entry point Skip it, or leave it to your platform's default, if: - The goal is rankings or citations in Google, ChatGPT search, or Perplexity - Nobody on the team will review the file when pages change - Your site is mostly marketing pages with no task an agent needs to complete If you skip it, nothing breaks. ![](https://storage.zerply.ai/teams/92/blogs/400/946591fa50134079-1791278770147-image.png) ## How do you write an llms.txt file? You write an llms.txt file by following the section order the proposal defines, then linking each entry to a Markdown version of the page it describes. The version 2 format, in order: 1. **An H1 with the site or project name.** This is the only required section. 2. **A blockquote** with a short summary containing what an agent needs to understand the rest of the file. 3. **Optional paragraphs or lists**, with no headings, giving context on how to read the links. 4. **H2 sections holding file lists.** Each item is a Markdown link, optionally followed by a colon and a short note. 5. **An H2 named "Optional"** for secondary links an agent can skip when context is tight. ### llms.txt example **Illustrative example.** Ledgerline is an invented invoicing API, and every URL is a placeholder on example.com. ```markdown # Ledgerline > Ledgerline is an invoicing API for B2B software companies. It creates invoices, collects card and bank payments, and syncs results to accounting tools. Prices on the pricing page are monthly, in USD, and exclude tax. API version 2026-08 is current. Version 2025-11 is deprecated and returns warnings. ## Docs - [Quickstart](https://example.com/docs/quickstart.html.md): API keys, first invoice, and test mode - [Invoices API](https://example.com/docs/invoices.html.md): Create, finalize, void, and list invoices - [Webhooks](https://example.com/docs/webhooks.html.md): Event types, retries, and signature checks ## Product - [Pricing](https://example.com/pricing.html.md): Plans, usage limits, and overage rates - [Security](https://example.com/security.html.md): Data residency and compliance reports ## Optional - [Changelog](https://example.com/changelog.html.md): Release notes by month - [Migration from 2025-11](https://example.com/docs/migrate.html.md): Breaking changes and code examples ``` Three choices make that file useful to an agent: - **The summary states what the product does in one sentence**, so an agent can decide in one read whether the file is relevant. - **The context paragraph answers the questions an agent would otherwise guess at**, such as billing currency and which API version is current. - **Every link points to Markdown, not HTML.** Version 2 proposes serving a clean Markdown copy at the same URL with `.md` appended, so `invoices.html` becomes `invoices.html.md`. ### Where should the file sit? The file can sit at the site root or at any path, and it covers the URLs under that path. A file at `/docs/llms.txt` covers everything in`/docs/`, and where two files apply, agents should use the more specific one. For a company whose agent-relevant content is all documentation, `/docs/llms.txt` is the better home. It keeps marketing pages out of a file meant for implementation questions. ## What are the llms.txt best practices? The llms.txt best practices that hold up are about getting the file found, keeping it small, and keeping it safe. - **Point agents to it.** Version 2 proposes a `rel="describedby"` link from each page to the llms.txt file that covers it, and a `rel="alternate" type="text/markdown"` link to the page's Markdown copy. Both can go in the page `` or in an HTTP header, as shown below. - **Mention it where agents get instructions.** OpenAI does this on its own crawler documentation, which opens by pointing readers to its llms.txt index and noting that `.md` versions of each page exist. - **Keep the file small.** The detail lives behind the links and gets fetched only when needed. Use the Optional section for anything an agent can skip. - **Treat it like code.** Ahrefs found a research crawler named `prompt-injection-survey` fetching llms.txt files, a sign that people are probing the file as an injection route into agents. Keep it in version control, limit who can edit it, write only plain links and descriptions, and link only to resources you control. - **Review anything auto-generated.** Platforms such as Wix, Yoast, and Mintlify can generate the file for you. Check what they publish before an agent reads it. - **Update it when the pages change.** An entry describing a deprecated API version sends agents to the wrong answer. - **Test it with an agent.** The proposal suggests giving an AI assistant only your llms.txt file and asking it questions about your product. Here is the header form for a single docs page, set in your server or CDN configuration so no page template changes: ```txt Link: ; rel="alternate"; type="text/markdown", ; rel="describedby" ``` ## How do you check whether anything reads your llms.txt? You check by filtering your server or CDN logs to requests for `/llms.txt`, then sorting those requests by bot type. Four questions turn the log into a decision: 1. **Did anything fetch the file this month?** If not, you are in the 97%, and further investment can wait. 2. **Which kind of bot fetched it?** Separate coding agents and assistants from training crawlers and SEO audit tools. Only the first group reflects the audience the file exists for. 3. **Are requests landing on a 200?** A server error is the one result Lighthouse flags, and it wastes every fetch that does arrive. 4. **Did agents follow the links?** Requests for your `.md` pages after a fetch of `/llms.txt` are the closest evidence that an agent used the file. Verify bot identity before trusting a user agent string, since scrapers copy the names of known bots. Then judge the file the way you would any AI visibility work, by its outcomes. Crawl requests show access. [AI referral traffic](https://zerply.ai/resources/blog/how-to-measure-and-act-on-ai-referral-traffic) and citations show whether anyone benefited. ## Conclusion Google settled the search question: llms.txt does nothing for rankings in Google Search, including AI Overviews and AI Mode. The open question is whether agents read your content, and for documentation-heavy sites, the answer can be yes. Publish the file when that audience exists, link agents to it, keep it as tightly controlled as code, and spend citation effort on the pages themselves, starting with our [answer engine optimization checklist](https://zerply.ai/resources/blog/best-practices-for-answer-engine-optimization). Zerply's [AI Traffic Analytics](https://zerply.ai/platform/ai-traffic-analytics/) answers the measurement question without a log parser. `/llms.txt` appears in its Crawled Pages view like any other URL, with the bots that requested it and how many times. Every visit is classified as AI Search, AI Training, or AI Assistant, so a training crawl of the file is easy to tell apart from an assistant fetch, and your `.md` pages show up in the same list when agents follow the links. Because the connection runs through Cloudflare, a site that does not use Cloudflare will need its own server or CDN logs for the same check. ## Frequently asked questions ### What is llms.txt? llms.txt is a Markdown file, usually placed at /llms.txt, that gives AI agents a short, curated map of a website with links to clean Markdown versions of its key pages. Jeremy Howard of Answer.AI proposed it in 2024, and version 2 of the proposal was published in August 2026. It is not a ratified standard. ### Does llms.txt help SEO or visibility in AI Overviews? No. Google's guide to generative AI features says Google Search does not use llms.txt, and that the file neither helps nor harms visibility or rankings in Google Search, including AI Overviews and AI Mode. Creating one for other services is fine. ### Do ChatGPT and Perplexity read llms.txt files? Rarely. In Ahrefs' study of 137,210 domains in May 2026, AI retrieval bots such as OAI-SearchBot and PerplexityBot made 1.1% of llms.txt fetches. OpenAI's crawler documentation does not say its crawlers read the file. Coding agents and training crawlers fetched it far more often. ### When is it worth publishing an llms.txt file? Publish one when coding agents or other AI agents need a guided path into your developer documentation, API reference, or product docs, and when someone will update the file as pages change. If your docs platform or CMS generates it automatically, review what it publishes before agents read it. ### Does Lighthouse require an llms.txt file? No. Lighthouse marks its llms.txt audit as not applicable when the file is missing. It flags a problem only when the server returns an error while fetching the file, so a broken llms.txt fails the audit while a missing one does not. ### How can I tell if anything reads my llms.txt? Filter your server or CDN logs to requests for /llms.txt, verify each bot against its operator's published IP ranges, and sort requests by bot type. Requests for your Markdown page versions after a fetch of /llms.txt are the closest evidence that an agent actually used the file. ```json {"@context":"https://schema.org","@type":"Article","headline":"llms.txt After Google's Guidance","description":"llms.txt guide for 2026: what Google says, who actually reads the file, a version 2 example, and best practices for publishing and measuring it.","url":"https://zerply.ai/resources/blog/llms-txt-guide","image":"https://storage.zerply.ai/teams/92/blogs/400/fb3cfeeeef308390-1791278624823-zerply-blog-banner-llms-txt-guide-2026-10-06_2x.png","datePublished":"2026-10-16T09:14:50+00:00","dateModified":"2026-10-16T09:14:50+00:00","author":{"@type":"Person","name":"Anshul Motwani"}} {"@context":"https://schema.org","@type":"FAQPage","mainEntity":[{"@type":"Question","name":"What is llms.txt?","acceptedAnswer":{"@type":"Answer","text":"llms.txt is a Markdown file, usually placed at /llms.txt, that gives AI agents a short, curated map of a website with links to clean Markdown versions of its key pages. Jeremy Howard of Answer.AI proposed it in 2024, and version 2 of the proposal was published in August 2026. It is not a ratified standard."}},{"@type":"Question","name":"Does llms.txt help SEO or visibility in AI Overviews?","acceptedAnswer":{"@type":"Answer","text":"No. Google's guide to generative AI features says Google Search does not use llms.txt, and that the file neither helps nor harms visibility or rankings in Google Search, including AI Overviews and AI Mode. Creating one for other services is fine."}},{"@type":"Question","name":"Do ChatGPT and Perplexity read llms.txt files?","acceptedAnswer":{"@type":"Answer","text":"Rarely. In Ahrefs' study of 137,210 domains in May 2026, AI retrieval bots such as OAI-SearchBot and PerplexityBot made 1.1% of llms.txt fetches. OpenAI's crawler documentation does not say its crawlers read the file. Coding agents and training crawlers fetched it far more often."}},{"@type":"Question","name":"When is it worth publishing an llms.txt file?","acceptedAnswer":{"@type":"Answer","text":"Publish one when coding agents or other AI agents need a guided path into your developer documentation, API reference, or product docs, and when someone will update the file as pages change. If your docs platform or CMS generates it automatically, review what it publishes before agents read it."}},{"@type":"Question","name":"Does Lighthouse require an llms.txt file?","acceptedAnswer":{"@type":"Answer","text":"No. Lighthouse marks its llms.txt audit as not applicable when the file is missing. It flags a problem only when the server returns an error while fetching the file, so a broken llms.txt fails the audit while a missing one does not."}},{"@type":"Question","name":"How can I tell if anything reads my llms.txt?","acceptedAnswer":{"@type":"Answer","text":"Filter your server or CDN logs to requests for /llms.txt, verify each bot against its operator's published IP ranges, and sort requests by bot type. Requests for your Markdown page versions after a fetch of /llms.txt are the closest evidence that an agent actually used the file."}}]} ```