What is llms.txt?

An llms.txt file is a plain text file placed in your website’s root directory (e.g., example.com/llms.txt) that provides AI systems with a structured overview of your content. Unlike traditional sitemaps, llms.txt uses Markdown formatting to organize information into meaningful sections, complete with context and summaries. Think of it as a semantic guide designed specifically for AI comprehension.

How does it help?

LLMs operate within limited context windows ¹ usually between 4,000 and 32,000 tokens depending on the model. This makes it hard to use LLMs to parse and provide usable information for large documentation sites. Though technically possible to process documentation directly from HTML, the presence of non-essential content (navigation elements, CSS, JavaScript, metadata and tracking scripts) add to the context window in lieu of actual content. Filtering out this ‘noise’ is resource-intensive and slows down the LLM’s performance.

llms.txt solves this by providing:

• A distilled, content-focused overview of your documentation
• Clear hierarchical relationships between different sections
• Concise summaries of each page’s key information
• Direct links to relevant content without surrounding noise

How is it different from sitemaps or SEO efforts?

Traditional sitemaps (sitemap.xml) help search engines find website pages, while SEO optimizes websites to rank higher in search results. llms.txt is designed specifically for LLMs, providing a structured, semantic summary of website content. While sitemaps aid discovery and SEO improves ranking, llms.txt enables AI to understand content. In an AI-driven future where LLMs power search and other applications, this semantic understanding becomes essential for accurate, relevant AI responses. Sitemaps and SEO still remain relevant for traditional search, but llms.txt is better positioned for an AI-centric future by prioritizing semantic understanding for LLMs.

Sample llms.txt file

# Website: TechProduct Documentation
## Purpose: Technical documentation for TechProduct API and features
 
### Getting Started
- /docs/quickstart
  Summary: Step-by-step guide for new users
- /docs/installation
  Summary: Installation instructions for all platforms
 
### API Reference
- /docs/api/v1
  Summary: Complete API documentation
- /docs/authentication
  Summary: Security and authentication guides
 
### Troubleshooting
- /docs/faq
  Summary: Common issues and solutions
- /docs/known-issues
  Summary: Current known issues and workarounds

This simple structure immediately tells an LLM:

• The website is about product documentation.
• There are key sections for getting started, API information, and troubleshooting.
• Each section links to relevant pages with helpful descriptions.

A few specific use-cases that an llms.txt file can be implemented :

• Creating an llms.txt file for a software API documentation: This file could list all the API endpoints, their parameters, and example usage, enabling LLMs to quickly answer developer questions about how to use the API.
• Developing an llms.txt file for a product’s user manual: This file could summarize key features, troubleshooting steps, and FAQs, allowing LLM-powered chatbots to provide instant support to users.
• Using llms.txt to guide LLMs in generating different versions of documentation: For example, you could use it to generate a quick start guide from a full user manual.

As a writer, I see some places where this can be very helpful:

• Content Findability for documentation sets, making it much easier for LLMs (and therefore users interacting with LLM-powered search or chatbots) to find the right information.
• Better Content Generation for LLMs can provide context and guidance to LLMs for generating documentation drafts, summaries, or translations.

Who’s Using It?

While still relatively new, llms.txt is starting to gain traction among forward-thinking organizations who understand the importance of preparing their content for AI like Anthropic (Claude AI) and Cursor (Cursor AI Code editor). llms.txt directory lists all sites that support the llms.txt standard.

How to add an llms.txt file to your site?

1. Analyze your website’s content: Identify key sections, topics, and entry points.
2. Create a plain text file named llms.txt: Use a text editor or code editor.
3. Use Markdown to structure your content: Provide a summary of the website and list important pages with descriptions.
4. Place the llms.txt file in the root directory of your website.
5. Maintain the file: Update it whenever your website’s content changes.

There is another file llms-full.txt that can contain all of your documentation as a single AI-consumable file.

There are several different tools you can use to create your own:

• Mintlify: generates both llms.txt and llms-full.txt files for hosted documentation.
• llmstxt by dotenv: generates llms.txt using your site’s sitemap.xml.
• llmstxt by Firecrawl: scrapes your website using Firecrawl to generate the llms.txt file.

How to use llms.txt with AI?

With ChatGPT and Claude, copy the contents of a site’s llms-full.txt file or the url into a chat and then proceed to ask specific questions.

You can also programmatically access the llms.txt or llms-full.txt and build prompts like:

• Using the documentation structure in llms.txt, where would I find information about API authentication methods?
• Based on the llms.txt structure, what prerequisites should I complete before starting the API integration?
• Using the content structure in llms.txt, provide a high-level overview of the available troubleshooting resources.

LLMs.txt represents a shift toward AI-first documentation by making documentation more accessible to AI systems while maintaining human readability.

Footnotes

1. The context window of LLMs is the number of tokens the model can take as input when generating responses. A larger context window is helpful in improving the usefulness of the LLMs. ↩