Your site can rank at position 1 in Google yet remain nearly invisible to AI systems like ChatGPT, Perplexity, and Claude. This discrepancy arises because AI language models do not interpret websites the same way Google does. They access content on demand, operate within limited context windows, struggle with JavaScript-heavy pages, and often miss important content that isn’t clearly linked or easily readable. The llms.txt file addresses this challenge.

Introduced in September 2024 by Jeremy Howard of Answer.AI and detailed at llmstxt.org, llms.txt is a plain Markdown file placed at your website’s root. It offers AI systems a curated, human-readable roadmap to your most vital content, eliminating guesswork about what matters on your site. By April 2026, leading companies like Anthropic, Stripe, Zapier, Cloudflare, Vercel, and Hugging Face had adopted it. However, only 5 to 15% of websites have implemented llms.txt, presenting a significant opportunity.

llms.txt is designed for AI models, unlike traditional web standards.

This guide covers what llms.txt is, how it differs from robots.txt and sitemaps, which frameworks support automatic generation, the precise format to follow, and how to create your own file in just 30 minutes. The free Claude SEO audit tool on this site already checks for llms.txt as one of its 27 signals. If your site didn’t pass that check, this guide will help you fix it.

What This Guide Covers

1. Key Takeaways

2. What llms.txt Actually Is

3. The 3-File AI Discovery Stack

4. Why llms.txt Exists: The AI Retrieval Problem

5. Who Is Already Using It

6. The Exact llms.txt Format

7. llms.txt vs llms-full.txt: Which Do You Need

8. Framework Support: WordPress, Yoast, Webflow, Next.js

9. How to Write Yours in 30 Minutes

10. The Honest Assessment: What It Does and Does Not Do

11. Where llms.txt Fits in Your Full GEO Stack

12. Frequently Asked Questions

Key Takeaways

• llms.txt is a plain Markdown file at your site root that gives AI language models a curated guide to your most important content. It does not affect Google rankings. It is designed entirely for AI retrieval systems.

• Only 5 to 15% of websites have implemented llms.txt as of early 2026, according to lowtouch.ai research. That adoption gap is the first-mover opportunity: sites that implement now are forward-compatible when AI platforms formalize support.

• Anthropic, Stripe, Zapier, Cloudflare, Vercel, Hugging Face, and a growing list of developer-tool companies have already implemented it. No major AI platform has officially confirmed automatic reading of llms.txt as a first-class input yet, but developer-facing tools including Cursor, GitHub Copilot, and RAG frameworks do read it when present.

• WordPress users can generate llms.txt automatically using Yoast SEO. Webflow provides a system to upload it directly to your root. Next.js, Mintlify, and Astro all have community plugins or built-in support.

• Creating a basic llms.txt takes under 30 minutes for most sites. The effort-to-potential-impact ratio is higher than any other single AI visibility action I have documented in two years of GEO work.

What llms.txt Actually Is

llms.txt is a plain-text Markdown file hosted at the root of your website, accessible at yourdomain.com/llms.txt. The llms.txt file should be located at the root path and hosted at /llms.txt. It offers a structured and concise overview of your site’s key content, tailored specifically for large language models to understand. llms.txt provides a curated overview for LLMs and acts as a curated guide for Large Language Models and AI agents. Serving as a curated guide for AI agents, it highlights the most important areas of your site.

A standard llms.txt includes the site name, description, key sections, and navigational links.

To visualize its role, imagine your website as a library: the sitemap.xml acts as the full card catalogue listing every book, robots.txt designates restricted sections, and llms.txt functions as the librarian’s recommended reading list, spotlighting the essential materials for a particular visitor. This file is designed to be easily interpretable by both humans and machines.

Unlike a sitemap, which is built for crawlers following links, llms.txt communicates in prose for AI systems. Unlike robots.txt, it doesn’t control access but suggests content priority. Unlike schema.org markup embedded within pages, llms.txt exists as a single domain-wide index, enabling AI systems to retrieve a clear, structured index in one request instead of crawling page by page. Typically, it includes the site name, description, key sections, and navigation links.

The specification was introduced in September 2024 by Jeremy Howard, co-founder of Answer.AI, and is maintained at llmstxt.org. The file must be named exactly llms.txt to ensure consistent AI discoverability across platforms.

The 3-File AI Discovery Stack

Understanding where llms.txt fits requires seeing all three files together. The diagram below illustrates how Google’s crawler, AI retrieval systems, and language models each utilize different files to comprehend your site.

The key distinction highlighted is that Google’s crawler reads robots.txt and sitemap.xml to enable search engines to discover pages and index your HTML content. AI systems like ChatGPT, Perplexity, and Claude also consult robots.txt for access permissions, but they face a fundamentally different challenge when accessing your content. HTML pages are designed for human readers, containing navigation menus, advertisements, JavaScript-rendered elements, and visual layouts. None of that structure helps an AI model extract the information it needs to answer a query. Real-world testing across Claude, ChatGPT, and Perplexity for core SEO tasks shows how differently each tool handles this constraint. llms.txt solves this by providing a clean, pre-organized summary that skips the noise entirely and can improve ai generated responses or AI-powered search results by pointing systems to high-value content.

Why llms.txt Exists: The AI Retrieval Problem for Search Engines

Traditional search crawlers operate through systematic, ongoing indexing. Googlebot regularly visits your site, stores the content it finds in a vast index, and retrieves information from that index when users perform searches. This process aims for thoroughness: Google endeavors to index all accessible pages over time.

In contrast, AI language models function differently. When a user queries ChatGPT or Perplexity, the system retrieves relevant web content on demand at that moment. It lacks persistent memory of your site from prior visits and is limited by context windows, meaning it can only process a finite amount of content per request. Additionally, AI models struggle with typical web page elements that Google handles easily, such as JavaScript-rendered content, complex navigation, and layout structures, which consume valuable processing space without providing meaningful information. This is why a Claude-based SEO checker workflow emphasizes clean, well-structured inputs over raw, cluttered HTML.

AI retrieval systems face specific challenges on standard websites: content rendered by JavaScript requiring browser processing before text is accessible; intricate HTML navigation and layout elements that waste context window space; important pages that are poorly linked internally, making discovery difficult; and a lack of signals indicating which content is most relevant for particular queries. The llms.txt file addresses these issues by guiding AI systems to answer questions more accurately and efficiently. A structured Claude SEO audit playbook for GEO and internal linking can surface these weaknesses quickly so your llms.txt reflects your strongest, cleanest assets.

A site with 500 pages and an excellent sitemap gives Google everything it needs. The same site gives an AI retrieval system a navigation problem: which of the 500 pages is most relevant to the query at hand? Without llms.txt, the AI has to guess based on its training data and whatever it can retrieve in the moment. With llms.txt, you tell it directly, steering retrieval away from irrelevant content and reducing token usage by up to 10x.

Who Is Already Using It

The adoption pattern as of April 2026 is revealing. Companies that implemented llms.txt first were primarily developer-focused companies whose target audience uses AI coding assistants: Anthropic itself (Claude’s documentation), Vercel (Next.js documentation), Stripe (API documentation), Cloudflare (developer documentation), Zapier (integration documentation), and Hugging Face (model documentation). These companies often publish structured developer docs and api references for AI use, so they benefit most immediately from llms.txt because their users regularly ask AI tools about their documentation, and accurate AI answers are a direct product support benefit.

The second wave of adoption is now underway among marketing websites, SaaS companies, and content publishers. Yoast SEO generating llms.txt automatically for WordPress sites is the signal that mainstream adoption has begun. When a plugin with millions of active installations includes a feature by default, that feature moves from early adopter to standard practice within 12 to 18 months.

For your specific situation: if your target clients use ChatGPT or Perplexity to research services in your category, the companies in your competitive set that implement llms.txt are giving AI systems a cleaner, more authoritative summary of their offering than your site currently provides, which can also drive referral traffic when your pages are surfaced in AI tools. That is a citation and recommendation gap that is straightforward to close.

The Exact llms.txt File Format

The official specification requires Markdown formatting with a specific structure. The llms.txt file must be formatted in Markdown. The file has three required sections and two optional ones. The first line of llms.txt must be an H1 heading. Here is the complete format with an explanation of each section.

Required Section 1: The H1 Title

The first line of the file is an H1 heading with your site or company name from the domain root on your own website. This is what AI systems use to identify whose content the file is describing, so the H1 should clearly match the source.

Kulbhushan Pareek , SEO and AI Marketing Consultant

Required Section 2: The Blockquote Description

Immediately below the H1, a blockquote provides a 2 to 3 sentence summary of what the site or company does. It works as a short description, with clear descriptions and contextual descriptions of what the site does, framing everything below it.

Digital marketing consultant with 13+ years helping US, UK, and European businesses build organic search visibility and AI search citation presence. Services cover SEO consulting, GEO optimization, AI marketing automation, and lead generation. Verified results: $385,091 organic revenue for a US B2B client in 18 months.

Required Section 3: H2 Sections With Links

The body of the file uses H2 headings to organize structured website information into logical categories, with each important page listed as a Markdown link with a brief description. The description after the link provides contextual descriptions that help ai models judge relevance. Treat this page list as structured content, and prioritize linking to canonical pages rather than duplicate or parameterized URLs.

Services

• SEO Consulting: Full-service SEO strategy and execution for US, UK, and European B2B companies. Technical audits, keyword research, content planning, link building, monthly GSC reporting.

• AI Marketing Automation: AI-powered marketing workflows replacing manual processes. Content production systems, automated reporting, competitor monitoring.

• GEO Optimization: Strategy for getting cited in ChatGPT, Perplexity, and Google AI Mode. GEO content restructuring, schema implementation, platform presence building.

Case Studies

• Indigo Software SEO Campaign: 18-month B2B SEO and AI visibility campaign. $385,091 verified organic revenue, 482% traffic growth, 369 AI-cited pages.

Key Guides

• 47 Claude SEO Prompts: Complete library of copy-paste Claude prompts for every SEO task including audits, keyword research, schema, and GEO optimization.

• LLM Citation Strategy: 5-method approach to building backlinks and platform presence that earns citations in ChatGPT, Perplexity, and Google AI Overviews.

• Technical SEO Audit Checklist: 12-point technical SEO checklist using Claude and free tools in 30 minutes. Includes Check 12 for AI crawler accessibility.

Tools

• Free Claude SEO Audit: Free 27-check SEO and GEO audit tool. Checks AI crawler access, llms.txt presence, and author schema alongside 24 classic SEO signals.

Contact

• Book a Free Strategy Call: Free 30-minute review of your SEO and AI search visibility.

• Email: Direct contact.

Optional Section: Notes on Sensitive or Excluded Content

You can add a final section noting what the llms.txt file does not cover and why. This is useful for sites with private documentation or content you want AI systems to handle carefully, but any sensitive areas still need separate access controls outside llms.txt so excluded material is not exposed. Read more about image SEO tool for AI search.

Notes

This file covers public-facing content only. Client reports and campaign data are confidential and not linked here.

Linking to Markdown Versions of Pages

The official specification recommends linking to pages’ markdown version URLs (ending in .md) rather than the HTML URLs where possible. Markdown files strip navigation, ads, and JavaScript and give AI systems clean, token-efficient content; in some cases, these md versions can reduce token usage by up to 10x. If your CMS generates Markdown versions of posts automatically (Mintlify, Ghost, and some Hugo configurations do), use those URLs. If not, your HTML URLs work fine. The specification permits both, but keep the main llms.txt index lightweight and ideally under 100 kilobytes.

llms.txt vs llms-full.txt: Which Do You Need

The llms.txt specification includes two distinct files with different functions.

llms.txt is the curated index: a file with links and short descriptions pointing to your most important pages. It usually follows a basic structure, while llms-full.txt is a different format meant for more exhaustive retrieval. It is typically short, between 200 and 2,000 words, and gives AI systems the map to your site. This is the file this guide has been describing. Almost every site should have this.

llms-full.txt is the complete content dump: a single Markdown file containing the full text of every page on your site flattened into one document. One site documented on Search Engine Land maintained an llms-full.txt file that was 115,378 words and 966KB. This format gives AI systems the entire content of a site in one request, eliminating the need to follow individual links, but it is not necessary for the entire site unless the whole website truly needs to be exposed as one machine-readable corpus. It was developed by Mintlify in collaboration with Anthropic and subsequently included in the official llms.txt proposal.

Who needs llms-full.txt: developer documentation sites where AI coding assistants need comprehensive API reference, platforms like Cursor and GitHub Copilot that fetch llms-full.txt to provide complete library context, and any site where the AI system needs to answer detailed technical questions from the full content body rather than just navigate to the right page.

Who needs only llms.txt: marketing websites, consulting sites, e-commerce sites, and blogs. For these sites, the curated index approach of standard llms.txt provides the right level of guidance without the server overhead of maintaining a massive single-file content dump.

For kulbhushanpareek.com and sites with similar structures, standard llms.txt is the correct choice. Create llms-full.txt only if you have developer documentation or a deep technical content library where AI assistants need full content access.

Framework Support: WordPress, Yoast, Webflow, Next.js

The framework support landscape for llms.txt has developed rapidly since early 2026. Here is the current status for the most common platforms.

WordPress with Yoast SEO

Yoast SEO now generates llms.txt automatically for WordPress sites, which is especially useful for documentation sites or sites with structured data needs because it helps publish organized website information automatically. The file is created from your published posts and pages, with your most recently updated content prioritized. Enable it in the Yoast dashboard under the SEO settings. The Yoast-generated file includes curated high-priority URLs, your latest blog posts, key documentation and important pages most relevant to LLMs, and optional link titles for clarity to help AI tools interpret page intent.

If you use Rank Math (which this site uses), a similar plugin is available from the Rank Math ecosystem. Check the Rank Math plugin directory for the current llms.txt module, as the feature set was being actively developed at publication time. As an alternative, the manual creation approach below works perfectly alongside Rank Math’s existing SEO functions.

Webflow

Webflow provides a native system to upload llms.txt directly to your root directory via the site settings. No custom code or developer involvement is required. Go to Site Settings, Publishing, and look for the File Manager or Custom Files section. Upload your llms.txt file there and make sure it is served from the root path so ai crawlers can fetch it reliably at yourdomain.com/llms.txt automatically.

Next.js

For Next.js sites, create a static file at public/llms.txt and it will be served at the root automatically. For dynamically generated llms.txt that updates as you add new content, create a route handler at app/llms.txt/route.ts that generates the file content from your CMS or content layer. The Vercel documentation at vercel.com/llms.txt shows their own implementation as a reference.

Mintlify

Mintlify generates both llms.txt and llms-full.txt automatically on every deploy. As a documentation platform, it is especially relevant for developer tools because AI coding tools rely on well-structured docs live. If you use Mintlify for documentation, your files are already being created. Check yourdomain.com/llms.txt to confirm they are live.

Hugo, Jekyll, and Static Site Generators

For static site generators, create a template file in your layouts directory that generates llms.txt from your content front matter. Hugo users can add llms.txt to their static directory for a manually maintained file or create a custom output format in config.toml for automatic generation. The community has published templates for both approaches in the Hugo forums, and reusable Claude Skills for automating SEO tasks can help keep those static builds aligned with how AI assistants actually consume your content.

PHP Sites (Custom CMS like kulbhushanpareek.com)

For PHP sites with custom CMS structures, the simplest approach is a static llms.txt file maintained manually and updated monthly, whenever major new content is published, or whenever the site’s content hierarchy or information architecture changes significantly. Create the file using any text editor, upload it to your site root (public_html/llms.txt), and verify it is accessible at yourdomain.com/llms.txt. This site uses this approach. The manual maintenance takes 5 minutes per update.

How to Write Yours in 30 Minutes

This is the step-by-step process for creating a complete, well-structured llms.txt file for most consulting, marketing, SaaS, or content sites. Total time: 25 to 35 minutes. You can pair this with a library of 47 Claude SEO prompts and tools to handle keyword research, content planning, and technical checks while you build out the file.

The llms.txt file should be located at the root path and hosted at /llms.txt.

Step 1: Open a Text Editor (2 minutes)

Use any plain text editor. Notepad on Windows, TextEdit on Mac (set to plain text mode, not rich text), VS Code, or any code editor. Do not use Microsoft Word or Google Docs unless you save as plain text, as rich text editors add formatting characters that corrupt Markdown.

Step 2: Write the H1 and Blockquote (5 minutes)

Your H1 is your business or personal brand name. Your blockquote is a 2 to 3 sentence description that answers: what do you do, who do you serve, and what is the one most important result or proof point. Keep the blockquote factual and specific. Avoid marketing language like “best in class” or “industry-leading.” Write as if briefing a researcher, not selling to a prospect.

Step 3: List Your Most Important Pages by Category (15 minutes)

Create H2 sections for each major content category: Services, Case Studies, Key Guides or Resources, Tools if applicable, and Contact. Under each H2, add a Markdown link for each important page with a one to two sentence description. This page list should function as structured content that helps AI agents navigate your most important website information. The description should answer what the page covers and who it is for, not repeat the page title.

Rules for page selection: include every service page, every case study, your 5 to 10 highest-traffic or highest-value blog posts, any free tools or resources, and your contact and booking pages. Do not include every blog post. The purpose is curation, not comprehensiveness. AI systems benefit from a prioritized list, not an exhaustive inventory. Each link should point to the most authoritative canonical page for that topic. Your sitemap already handles the exhaustive inventory.

Step 4: Add the Notes Section (3 minutes)

Add a brief Notes section if your site has content you want AI systems to understand is excluded. For most sites this is private client data, confidential reports, or draft content. One or two sentences is sufficient.

Step 5: Save as Plain Text and Upload (5 minutes)

Save the file as llms.txt (not llms.md or llms.txt.txt). Upload it to your site root via FTP, SFTP, or your hosting control panel’s file manager. For cPanel-based hosting (like Hostinger VPS), navigate to File Manager, go to public_html, and upload the file directly there.

Step 6: Verify and Submit (5 minutes)

Open a new browser tab and go to yourdomain.com/llms.txt. You should see your plain Markdown text rendered in the browser. If you see a 404 error, the file is not in the correct directory. If you see formatted text (bold, headers visually rendered), the file is being served correctly and the browser is interpreting the Markdown, which is fine.

After verifying, go to Google Search Console, use URL Inspection on yourdomain.com/llms.txt, and click Request Indexing. Do the same in Bing Webmaster Tools. Also submit the URL to your XML sitemap as an additional page so it gets discovered consistently. To confirm actual AI crawler requests, review your server logs for user-agents such as Google-Extended and check whether they are requesting llms.txt. Once Search Console is connected, a direct GSC-to-Claude integration via MCP lets you query this crawl and index data in natural language alongside your llms.txt improvements.

Step 7: Run the Free Audit to Confirm (2 minutes)

Go to the free Claude SEO audit tool and run your site or use the standalone AI SEO + GEO audit tool with 31 checks. The llms.txt check in the audit will now show as passed. If it still shows as failed, clear your browser cache and run again, or check that the file is genuinely accessible at the root URL (not in a subdirectory).

The Honest Assessment: What It Does and Does Not Do

Every guide on llms.txt should include an honest section on what is confirmed versus what is speculative. The research is clear, and practitioners deserve the full picture rather than inflated promises.

What Is Confirmed

Developer-facing AI tools do read llms.txt. Cursor, GitHub Copilot, Continue, Aider, and various RAG (Retrieval-Augmented Generation) frameworks fetch and use llms.txt when it is present. For sites with a developer audience, this is an active and verifiable benefit.

AI retrieval pipelines can be prompted to fetch it. When users or developers ask ChatGPT, Claude, or Perplexity to read your llms.txt as part of a workflow or research task, the file provides structured, clean information that those systems would otherwise have to scrape from your HTML pages. This is a practical user-experience benefit for anyone using AI assistants to research your site.

It provides a clean content signal. In developer workflows, it can reduce token usage by up to 10x when AI assistants fetch cleaner sources, which also helps coding tools work with less noise and more relevant context. Even if no major AI platform reads it automatically yet, llms.txt creates a clean, token-efficient summary of your site that reduces the noise-to-signal ratio when an AI system does encounter your content. Sites with llms.txt are forward-compatible for when platform support formalizes.

What Is Speculative or Unconfirmed

No major AI platform has officially confirmed automatic reading of llms.txt as a first-class input, and no major ai company has publicly committed to the standard that way. As of April 2026, Anthropic, OpenAI, and Perplexity have not published statements committing to read it automatically. Google’s John Mueller has noted that major crawlers do not currently prioritize these files over standard HTML. Semrush research found no statistical correlation between implementing llms.txt and improved performance in AI results in their controlled study.

The absence of confirmation is not evidence of no value. Web standards typically follow the pattern that sites publish first and platform support formalizes after adoption crosses a threshold. Robots.txt existed as a convention before any search engine officially committed to respecting it. The adoption curve for llms.txt is tracking a similar pattern, with developer tool adoption leading and AI platform adoption following.

The honest conclusion: llms.txt is a low-effort forward investment with low implementation cost and minimal downside. Creating it takes 30 minutes. It provides confirmed value for developer tool AI workflows today and positions you for the emerging mainstream AI platform support. If it never becomes a universally consumed standard, you still have a clean, text-only summary of your brand that remains easily readable by both humans and AI agents when prompted. That has independent value regardless of automatic adoption.

Where llms.txt Fits in Your Full GEO Stack

llms.txt is one technical signal in a broader AI search visibility strategy. Understanding where it fits prevents the mistake of treating it as a complete GEO solution rather than one layer of a multi-signal approach. It complements traditional SEO rather than replacing it, and its role is improving AI visibility and retrieval clarity rather than directly influencing search rankings.

The GEO stack has five layers, each addressing a different aspect of how AI systems discover, evaluate, and cite your content. llms.txt addresses discovery: helping AI systems find and understand your content. Structured website information in this layer can support cleaner AI retrieval and stronger AI-powered search behavior. The other four layers address content structure (GEO-ready formatting with direct-answer openings and named statistics), technical signals (schema markup, AI crawler access in robots.txt, Bing Webmaster Tools verification), entity authority (LinkedIn publishing, Reddit presence, platform citations on G2 and Clutch), and brand mentions (expert publication placements, HARO responses, earned citations). A 90-minute Claude-led SEO audit workflow is a practical way to keep these layers technically healthy over time. All five layers are required for consistent AI citation frequency. llms.txt alone does not produce citations. It removes one technical friction point in the path to being cited.

For the complete GEO implementation guide covering all five layers, the GEO and AEO optimization guide covers the full technical and content approach. For building the platform presence that produces AI citations at scale, the LLM citation and backlink strategy guide covers the complete 5-method approach with verified campaign results.

The technical foundation layer (which llms.txt is part of) also includes the AI crawler accessibility check from the 12-point technical SEO audit checklist. If your robots.txt is not explicitly allowing ChatGPT-User, PerplexityBot, and ClaudeBot, fix that alongside creating your llms.txt file. While it is separate from standard search engine optimization files, both support discoverability in different systems. Both changes take under 30 minutes combined and together address the two most commonly missed technical AI visibility prerequisites.

For the consulting service that implements the full GEO stack as an end-to-end engagement, the SEO consulting service covers GEO optimization as a core component of every engagement since the AI search shift became measurable in campaign data. You can also book a free 30-minute SEO strategy call if you want a live walkthrough of how llms.txt fits your specific site. The verified results from implementing this full stack are documented in the Indigo Software case study and the broader library of case studies and results-focused SEO guides: 369 AI-cited pages, 534 brand mentions, and $385,091 in organic revenue attributed over 18 months.

Captions

How does llms.txt relate to AI Search Visibility?

AI Search Optimization, Discovery, Clarity and Guidance. While this concept is often mentioned within the GEO space, there’s one file format that goes hand-in-hand.

llms.txt. (get your AI crawlers out of the JavaScript! )

According to 2026 research from lowtouch.ai, only 5 to 15% of websites have implemented llms.txt, presenting a significant first-mover opportunity.

Building an llms.txt file that AI systems like ChatGPT, Perplexity, and Claude know, understand, and trust does as much for your AI search reputation as your traditional SEO one.

Want to learn how? We’ve got details for you below!