Quick Summary: LLMS.txt is a plain-text Markdown file at your site root that gives large language models a clean, curated map of your most important content. It cuts the noise of HTML, CSS, and JavaScript, and is quickly becoming the preferred format for AI agents and coding tools to understand websites efficiently.
Introduction
Websites are built for humans and traditional search engines. That means heavy HTML, navigation chrome, ads, scripts, and styling. Large language models (LLMs) and AI agents, however, work best with clean, structured text that fits inside limited context windows. Parsing full rendered pages wastes tokens and often loses the signal.
LLMS.txt solves this. Proposed by Jeremy Howard in September 2024 and documented at llmstxt.org, it is a simple Markdown file served at /llms.txt. It provides a high-signal summary of your site plus annotated links to the pages that matter most. No rendering, no CSS, no JavaScript—just text that both humans and models can read.
Right now, traffic is still light. Many site owners report that the main visitor to their LLMS.txt is Grok (from xAI) along with a handful of smaller bots and coding agents. That early signal is exactly why the format is gaining traction so fast: the markup is already the language LLMs prefer to consume.
Prerequisites
Before creating or serving an LLMS.txt file, make sure you have:
- Write access to the root of your website (or the ability to place a static file at the domain root)
- Basic familiarity with Markdown syntax
- A clear list of your highest-value pages (docs, product pages, about, pricing, key guides)
- Optional: a Markdown editor or simple text editor and a way to deploy static files
How do you create a basic LLMS.txt file?
Start with the required structure defined by the proposal. The file must begin with a single H1 containing the project or site name. Follow it with an optional but strongly recommended blockquote summary, then free-form paragraphs if needed, and finally H2 sections that group Markdown links.
# Your Project or Site Name
> One or two sentence summary of what the site or project does and who it is for.
Optional extra context paragraphs can go here. Keep them short and factual.
## Documentation
- [Getting Started](https://example.com/docs/getting-started): Installation and first steps
- [API Reference](https://example.com/docs/api): Full endpoint documentation
## Optional
- [Changelog](https://example.com/changelog): Release historyExplanation of the Code
- H1 heading: The only strictly required element. It names the site or project and is the first thing a model sees.
- Blockquote summary: Gives the model an immediate, quotable description. Models often extract this line verbatim.
- H2 sections + links: Each link uses the form
- [Title](url): short description. The description helps the model decide whether to fetch the linked page. An## Optionalsection signals lower-priority content that can be skipped when context is tight.
Comparison of Available Options
| Option | Format | Audience | Overhead | Primary Use Case |
|---|---|---|---|---|
| Full HTML pages | HTML + CSS + JS | Humans + search crawlers | High (tokens + parsing) | Visual browsing and traditional SEO |
| sitemap.xml | XML | Search engine crawlers | Medium | Complete URL inventory |
| robots.txt | Plain text | All crawlers | Very low | Access control (allow/disallow) |
| LLMS.txt | Markdown | LLMs and AI agents | Extremely low | Curated, high-signal map for models |
| LLMS-full.txt | Markdown | Agents needing full corpus | Higher (full content) | One-shot ingestion of entire documentation |
LLMS.txt sits in a unique spot: it is selective rather than exhaustive, and it is written in the exact format models already process well.
Step-by-Step Configuration Guide
- Decide what belongs in the file: List the 10–50 pages that best represent your site—core docs, product overview, pricing, about, key guides. Avoid dumping every URL.
- Create the file: Write the Markdown following the structure above. Keep the total size modest (many real-world files sit under 20 KB).
- Place it at the root: Upload or deploy the file so it is reachable at
https://yoursite.com/llms.txt. Some documentation platforms also support/docs/llms.txt. - Optionally add an LLMS-full.txt: For documentation-heavy sites, create a second file that concatenates the full Markdown content of key pages. This is useful for agents that want the entire corpus in one fetch.
- Verify and monitor: Visit the URL in a browser to confirm it serves correctly as plain text. Check your server logs for requests. Early adopters often see Grok and coding-agent traffic first.
- Keep it current: Update the file when major pages change or new high-value content is published. Stale links reduce usefulness.
Frequently Asked Questions (FAQ)
What causes LLMS.txt to be ignored by most crawlers today?
Most major AI search systems still rely on their existing HTML pipelines and have not publicly committed to reading the file. Early traffic is dominated by coding agents (Claude Code, Cursor, etc.) and a few experimental bots. That is expected for a young convention—adoption by agents tends to lead broader usage.
Can this approach be used in production environments?
Yes. The file is static, costs almost nothing to serve, and does not interfere with robots.txt or sitemaps. Many documentation sites (Anthropic, Cloudflare, Stripe, OpenAI docs, and others) already publish one. Treat it as low-effort content hygiene that positions you for agentic tools.
Why is pure Markdown better than HTML for LLMs?
HTML carries layout, scripts, styles, and navigation that consume tokens and dilute the signal. Markdown is already the native format many models are trained to reason over. There is no rendering step, no CSS or JavaScript to strip, and the structure (headings, lists, links) maps cleanly to how models parse information.
Is LLMS.txt the same as robots.txt?
No. robots.txt controls access (what may be crawled). LLMS.txt is a curated invitation: here are the pages that matter and what they contain. The two files complement each other.
Conclusion
LLMS.txt is still early. Today the primary visitors are Grok and a small set of coding agents and minor bots. That limited traffic is precisely the signal that the format is working: it is already the clean, low-overhead language that language models prefer. As more agents and tools look for structured, token-efficient entry points, sites that publish a well-maintained LLMS.txt will be easier for models to understand and cite accurately.
Create the file, keep it focused, and treat it as the machine-readable front door to your content. The cost is near zero; the upside compounds as the agentic web grows.
