What Is llms.txt and Do I Need One?

llms.txt is a proposed Markdown file—living at https://yoursite.com/llms.txt—that AI agents can read to discover page URLs, key documentation paths, or API endpoints. Jeremy Howard introduced the idea on X, inspired by humans.txt. A growing list of early adopters (HuggingFace, Anthropic, LangChain, and others) now serve llms.txt files. Google Search explicitly said in late November 2024 that Google Search “does not use llms.txt” and has no plans to. This guide explains what llms.txt is, who says they read it, what Google said, and when you might still want one.

For a free, no-signup pass focused on SEO checks (including sitemap, canonical, and robots signals), AI-crawler / robots readiness, and a prioritized fix list, run your URL at SiteScope. An experimental AI-knowledge check may appear on some reports; it runs on Cloudflare Workers AI (Llama 3.1 8B), uses learned knowledge—not live ChatGPT or Perplexity answers—and is not guaranteed.

What llms.txt is

Jeremy Howard described llms.txt as a way to help LLMs (Large Language Models) and agents find relevant docs, similar to how robots.txt guides crawlers or humans.txt lists team members. The minimal proposal:

1. Put a Markdown file at https://yoursite.com/llms.txt (served as text/plain).
2. Include a short summary of what the site offers, then list helpful paths—documentation, API references, key pages.
3. AI agents or LLM-powered tools can optionally retrieve it to understand site structure before crawling or answering user questions about your product/company.

The proposal lives at llmstxt.org, which aggregates examples and explains usage. The format is intentionally lightweight—not an official standard—so there is no rigid spec, no validator, and no published list of which models actually fetch it in practice.

Who adopts it (examples from early adopters)

As of late 2024, a small but notable group of dev-tool and AI-research organizations serve llms.txt files:

• HuggingFace — lists their main docs and model hub sections. (https://huggingface.co/llms.txt)
• Anthropic — includes links to Claude API documentation, safety guidelines, and research papers. (https://www.anthropic.com/llms.txt)
• LangChain — points to their Python and JavaScript docs, cookbook, and integrations list. (https://www.langchain.com/llms.txt)
• fast.ai (Jeremy Howard’s own project) — links to course notebooks and the fastai library docs. (https://www.fast.ai/llms.txt)
• Vercel, Replit, and others — some larger dev platforms have also shipped files after the proposal gained traction on X and HackerNews.

Because it is voluntary, you cannot assume every AI or agent actually requests llms.txt yet. OpenAI ChatGPT browsing, Claude Computer Use, or Perplexity citations do not officially advertise guaranteed support—though individual models or tools may fetch the file opportunistically.

Google Search explicitly ignores it

On November 26, 2024, Google’s SearchLiaison account confirmed on X: “We don’t use llms.txt and have no plans to.” (X post) That was in response to a user question about whether llms.txt affects Google Search rankings or AI Overview eligibility.

This means:

• Adding llms.txt will not help your site rank in Google Search results.
• It will not make your pages more likely to appear in AI Overviews or AI Mode answers inside google.com search.
• Google Search already has sitemap.xml, structured data, and clear crawl guidelines—llms.txt adds no recognized signal for Google’s web index.

Some site operators hoped that a well-structured llms.txt might influence how Gemini models answer questions outside of Search (in the standalone Gemini chat interface, or in Google AI Studio). Google has not confirmed whether those surfaces read llms.txt either. For now, treat Google’s statement as covering Google Search specifically.

Why Google does not need it

Google Search already relies on:

• Sitemap.xml for URL discovery.
• robots.txt for crawl permissions (including AI bots like Google-Extended for optional Gemini training data blocking).
• Structured data (JSON-LD / Microdata) for rich-result eligibility.
• Internal link hierarchy and page quality signals for ranking.

An extra Markdown file describing the same docs would be redundant for Google’s crawler. Google’s SearchLiaison comment suggests they view llms.txt as solving a problem Google already solved with existing specs.

When to add llms.txt anyway

You might still serve llms.txt if:

• Developer / API docs site — your audience includes AI researchers or agent developers who might actually look for llms.txt to speed up integration or citation of your API.
• Early-adopter signaling — publishing llms.txt can be a lightweight way to show your docs are “LLM-friendly.” The cost is near zero (one text file), and if more AI tools start reading it, you are already compliant.
• Agent-first use case — if your product is built around AI agents (e.g., a tool docs site for LangChain or AutoGPT users), having llms.txt aligns with community expectations.
• Minimal maintenance — unlike ongoing SEO or structured-data work, llms.txt is typically a one-time publish with infrequent updates. If you already have a clear site outline, adding it takes minutes.

Do not add llms.txt expecting Google Search ranking gains, faster indexing, or guaranteed usage by ChatGPT or Claude. The value is speculative for most sites—aimed at a future where more agents adopt the convention, not at today’s mainstream search engines.

How to create one (minimal example)

The llms.txt file is plain Markdown served at the root of your domain with Content-Type: text/plain. Do not make it HTML; keep it machine-readable text.

Step 1: Create the file

Place it in your public/ directory (Next.js, Vite) or equivalent static root so it is accessible at https://yoursite.com/llms.txt.

Step 2: Add a short summary + links

Example for a docs site:

# MyProduct Documentation MyProduct is a developer tool for X. Here are the main docs and API references. ## Key Pages - Documentation: https://yoursite.com/docs - API Reference: https://yoursite.com/api - Guides: https://yoursite.com/guides - Support: https://yoursite.com/support

Keep it concise. List the URLs an agent would want; skip footer links, marketing pages, or deep paths that are already linked from those top-level docs.

Step 3: Serve as text/plain

Most static hosts (Cloudflare Pages, Vercel, Netlify) serve .txt files as text/plain by default. Confirm with curl -sS https://yoursite.com/llms.txt—you should see plain Markdown, not an HTML wrapper.

Step 4: Optional robots.txt pointer (not required)

Some early examples added a comment in robots.txt pointing to llms.txt, e.g.:

User-agent: * Allow: / # LLM agents can see llms.txt for site overview # https://yoursite.com/llms.txt

This is not mandatory—agents that support llms.txt will check the root path directly. The comment is a courtesy note for humans inspecting robots.txt.

Checking if your file is live

1. Browser — visit https://yoursite.com/llms.txt and confirm it loads as plain text (not a 404).
2. curl — curl -sS https://yoursite.com/llms.txt should return Markdown without HTML tags.
3. Content-Type header — curl -I https://yoursite.com/llms.txt should show Content-Type: text/plain (or text/markdown—both work).
4. llmstxt.org — the proposal site has a simple checker; paste your domain to verify format.

Once live, the file is passively discoverable. No ping or submission step is needed—unlike sitemap submission to Search Console.

FAQ

Does llms.txt improve Google rankings?
No. Google Search explicitly said they do not use it and have no plans to. (SearchLiaison on X)

Will ChatGPT or Claude definitely read it?
Neither OpenAI nor Anthropic has published official support guarantees for llms.txt. Anthropic’s own site has one, suggesting internal teams find it useful, but that is not a promise that Claude always fetches external llms.txt files during browsing.

Is there an official spec?
No. The proposal at llmstxt.org is community-driven and voluntary. The format is Markdown with no rigid validator.

Should I replace my sitemap with llms.txt?
Absolutely not. sitemap.xml is the standard for search engines (Google, Bing) and is far more important for SEO. Keep your sitemap; llms.txt is supplementary for AI agents, not a replacement.

Can it hurt?
Serving a plain-text file at /llms.txt has no known SEO downside. The risk is near zero—worst case, no one reads it.

Soft next step

If your site targets AI researchers, developer tool users, or early-adopter communities, add a minimal llms.txt linking to your docs, API, and guides. For mainstream sites focused on Google Search rankings, prioritize sitemap.xml, structured data, and robots.txt AI-bot rules instead—those have documented impact. For a free, no-signup report focused on SEO checks, AI-crawler / robots readiness, and a prioritized fix list (plus optional comparison against up to three competitors), run your URL at https://sitescope.pages.dev. An experimental AI-knowledge check may appear on some reports; it is not guaranteed and is not a live ChatGPT or Perplexity answer readout.

Sources

• Jeremy Howard — Original X thread introducing llms.txt (November 2024)
• Google SearchLiaison — “We don’t use llms.txt and have no plans to” (November 26, 2024)
• llmstxt.org — community proposal site with examples (HTTP 200)
• Early adopter examples — HuggingFace, Anthropic, LangChain, fast.ai (all HTTP 200 at time of writing)
• SiteScope — sitescope.pages.dev (HTTP 200)