llms.txt in Practice: Author One That Improves AI Retrieval

Your competitors are getting cited by ChatGPT, Perplexity, and Gemini while your content sits ignored. You check your AI search visibility and see zeros across the board. Traffic that used to flow from organic search is leaking to AI answer engines, and your pages aren't in the conversation. The fix starts with a single llms txt file.

This file tells large language models which pages matter, what your site covers, and how to prioritize content for retrieval. It’s not a robots.txt - it’s a handshake with AI crawlers. If you ignore llms.txt, you’re leaving citations on the table. If you implement it well, you give assistants like ChatGPT, Perplexity, Claude, and Grok a map to your authoritative content.

In this guide, you’ll build an llms.txt from scratch. We’ll cover prerequisites, a step-by-step authoring process, validation, and the monitoring loop that catches whether the file actually improves AI retrieval. No hand-waving. Concrete, operator-level tactics.

Prerequisites

Before you open a text editor, get these foundations right:

• AI‑ready content. llms.txt points to pages, but those pages need to be answer‑engine optimized. If your content is thin, outdated, or lacks entity‑rich structure, even a perfect llms.txt won’t make it citable. Start by auditing your AI citation performance and applying 5 content strategies that work for AI search.
• Site ownership. You must be able to place the file at https://yourdomain.com/llms.txt. If you’re behind a CMS that doesn’t let you surface plain‑text files at the root, fix that first. For example, Shopify stores can use schema and structured data plugins to serve equivalent markup, but a standalone llms.txt is ideal.
• A content inventory. Know which URLs actually rank, convert, or demonstrate topical authority. You’ll select up to 30 URLs for the file, so you need this list curated.
• AI crawler awareness. Understand the user agents you’re targeting: GPTBot, PerplexityBot, Google-Extended, anthropic-ai, Claude-Web. Not all of them read llms.txt, but those that do will treat it as a canonical hint.

Pro tip: If you haven’t yet built an AEO workflow inside Claude, doing so now lets you automate monitoring later - the SearchFit MCP tools can watch for citation changes after you publish the file.

Step 1: Understand What llms.txt Is - And What It Isn’t

llms.txt is a plain‑text file that provides contextual information and curated links to language models. Proposed by Jeremy Howard in 2024, it follows a Markdown‑like format that’s easy for both humans and machines to parse. GitBook’s explanation nails the basics: it’s a way to present canonical content, reducing hallucinations and improving retrieval relevance.

But there’s a nuance. Contentful’s analysis points out that no major AI provider has officially endorsed llms.txt as a ranking signal, and some crawlers ignore it. That’s true. However, the same analysis acknowledges that when assistants do read the file, they leverage it to weight pages differently. I’ve seen sites with a clean llms.txt see a measurable uptick in citation frequency - not because the file magically boosts rankings, but because it removes ambiguity. The model doesn’t have to guess which page is the best definition of “churn prediction”; you tell it.

Think of llms.txt as a digest that you control. It’s particularly powerful when combined with structured data and consistent brand mentions. AI models use entity signals, and llms.txt reinforces your entity graph.

Step 2: Curate the Right Content Inventory

Most llms.txt implementations fail because they dump every URL or pick pages at random. Don’t do that. AiRankLab’s 2026 best practices recommend a maximum of 30 URLs, ordered by importance. Their research shows that models truncate long lists, so your top 10 links get disproportionate attention.

To build your inventory:

1. Pull your high‑authority pages. Export your top‑performing pages from Google Search Console and GA4. If you’re running an AEO content brief template, you’ll already have a list of pages that target AI‑answerable queries.
2. Filter for answer‑ready content. Only include pages that directly answer a question a user might ask an AI assistant. Product pages usually don’t qualify unless they’re detailed buying guides. Blog posts, guides, definition pages, and case studies do.
3. Group by user journey. Instead of a flat list, use logical sections (I’ll show the format in Step 3). For example: “Core concepts,” “Implementation guides,” “Case studies.” This mirrors how AiGrowthAgent advises grouping to match user intent patterns.
4. Exclude thin, outdated, or paywalled content. If a page has <300 words, a last‑modified date older than 18 months without updates, or a hard paywall, leave it out. You can use a companion llms-full.txt for deeper indexing, but that’s a separate file for models that can ingest larger context windows.

Warning: Overstuffing the file dilutes the signal. The model’s attention mechanism discounts long, noisy prompts. If you include 200 URLs, expect maybe the first 10-15 to get any weight.

Once you have your inventory, validate it against your SEO goals. If your AI visibility tracking shows zero citations for “topic clusters,” the pages you select should directly address those clusters.

Step 3: Structure the llms.txt File for Maximum AI Retrieval

The official spec is loose, but the community has converged on a best‑practice structure. Derivatex Agency’s 2026 guide outlines four elements: an H1 heading, a blockquote summary, section headings, and annotated links. Limy’s full guide adds structural rules and a testing framework. CrawlSense’s implementation guide suggests optional schema.org markup and read‑side instrumentation.

Here’s the template I use, optimized for retrieval by ChatGPT, Perplexity, Gemini, and Claude:

# [Your Site Name]

> [A one‑paragraph summary of what your site covers, in 2-3 sentences. Include key entities and your primary topic. This acts as a system prompt for the model when it opens your file.]

## [Section Name 1]

- [Page Title 1](https://yoursite.com/page1): Brief, descriptive annotation - explain what the page teaches and why it’s authoritative.
- [Page Title 2](https://yoursite.com/page2): Another annotation.

## [Section Name 2]

- [Page Title 3](https://yoursite.com/page3): Annotation.
...

Key rules:

• Use a single H1 (#) for the site title.
• Use a blockquote (>) for the summary - this gets parsed as a context snippet.
• Use H2 (##) for sections. H3s are optional but can help group sub‑topics.
• Each link is a Markdown list item with an annotation. The annotation is critical: it tells the model exactly what the page covers and its relevance. Without annotations, you’re just a link dump.
• Limit to 30 links total, with at most 10 per section if you have multiple sections.

Pro tip: The annotation should mirror the query intent you want the model to associate. If you want ChatGPT to recommend your “churn prediction guide” when a user asks “How to reduce SaaS churn,” write the annotation as: “Comprehensive guide to churn prediction and reduction tactics for B2B SaaS - includes frameworks, benchmarks, and tool recommendations.”

Step 4: Author the File - Line by Line

Now let’s write it. I’ll use a fictional B2B SaaS site as an example, but you’ll adapt to your domain.

1. The H1: Use your brand or site name. Don’t overthink it. # Acme Analytics works. If you have a tagline that clarifies your niche, you can add a subtitle, but it’s unnecessary.

2. The blockquote summary: This is your chance to shape the model’s understanding. Make it concise and entity‑dense. Example:

Acme Analytics is the technical authority on B2B SaaS churn prediction, retention metrics, and customer health scoring. Our guides, benchmarks, and case studies help revenue teams reduce churn by 15‑30% using data‑driven playbooks.

Notice I didn’t fabricate the 15‑30% figure - that’s a placeholder. Replace with a real data point you’ve published. If you don’t have one, use a qualitative statement: “helps revenue teams measurably reduce churn.”

3. The sections and links: Group your URLs into logical buckets. For a SaaS, you might have:

• Core Concepts

• Implementation Guides

• Advanced Analytics

• Case Studies

For each, add no more than 5-7 links with annotations. Example:

## Core Concepts

- [What Is Churn Prediction?](https://acme.com/blog/churn-prediction): Foundational article explaining churn prediction models, key metrics (activation, MRR churn), and how to set up a prediction pipeline.
- [Customer Health Scoring Explained](https://acme.com/blog/customer-health-scoring): Detailed breakdown of health‑score components, how to weight attributes, and how leading SaaS companies use health scores.

4. Annotations that work: The best annotations match the long‑tail queries you want to rank for in AI answers. If you have a page targeting “AI visibility for Shopify,” your annotation might be: “Step‑by‑step implementation of product schema and structured data for Shopify stores to appear in AI answers - includes plugin walkthrough.” That’s exactly how I’d annotate that resource.

5. The companion llms-full.txt (optional): For models with large context windows (like Gemini 2.0 Pro or GPT‑4‑turbo), you can serve a longer version. This file should contain the full text of your most important pages, concatenated in Markdown. Limy’s guide recommends keeping this under 1MB and updating it weekly. I’ve seen this work particularly well for documentation‑heavy sites. If you run a marketing data lake that feeds content, you can automate the assembly of llms-full.txt from your content warehouse.

Pro tip: Don’t publish llms-full.txt unless you’re confident your content is clean and authoritative. A poorly maintained full‑text file introduces noise and can confuse the model.

Step 5: Validate and Test Your llms.txt

Without testing, you’re guessing. Here’s a three‑stage validation process:

5.1 Syntax and accessibility check

• URL access: Open https://yourdomain.com/llms.txt in a browser. You should see the raw text, no HTML wrapping. If your CMS adds a template, fix it - the file must be served as text/plain.
• Markdown lint: Use a markdown linter to catch broken links or formatting errors. Even a missing closing bracket can throw off parsers.
• robots.txt alignment: Ensure your robots.txt doesn’t block the AI crawlers you’re targeting. Some assistants will still read llms.txt even if disallowed, but it’s safer to allow. Check with Google’s robots.txt tester.

5.2 AI session testing

This is the true test. Open a fresh chat with each assistant - ChatGPT, Perplexity, Gemini, Claude - and ask questions that should trigger citations from your site. Watch whether the model references your pages. You’re looking for two things:

• Citation frequency: How often does your site appear in answers?
• Citation accuracy: Is the model pulling the correct page and representing it accurately?

If you’ve built an AEO agent team, you can automate this with a scraping pipeline. The SearchFit Claude plugin with MCP does exactly this - it runs queries across engines and reports back which pages were cited.

5.3 Instrumentation (optional but recommended)

Add UTM‑like tracking to your llms.txt links to measure downstream traffic. For example:

- [Guide](https://acme.com/guide?ref=llmstxt)

Then filter that parameter in GA4 to see if assistants actually click through. This isn’t perfect - many assistants don’t click links - but it gives a floor. CrawlSense’s guide mentions read‑side instrumentation that can track when the file is fetched, which is even better.

Warning: Don’t rely on assistant‑side behavior for validation. Some assistants cache the file, so changes may take days to propagate. Retest after 48 hours.

Step 6: Monitor AI Citations and Iterate

Authoring the file is just the beginning. The real work is watching how it changes your AI retrieval performance and continuously refining.

6.1 Set up AI citation tracking

Use a tool like SearchFit’s AI tracking to monitor your brand mentions across ChatGPT, Perplexity, Gemini, Claude, and Google AI Overviews. You need to see:

• Which pages are getting cited
• How often
• For which queries
• How your share of voice stacks up against competitors

If you’re a B2B SaaS company, you can model the approach from this case study that increased AI citations by 340% by pairing llms.txt with answer‑engine optimization.

6.2 Compare pre‑ and post‑llms.txt data

If you didn’t take a baseline before publishing, do it now. Pull your AI citation data for the 30 days before launch and compare to the 30 days after. Look for:

• New pages appearing: Are pages that weren’t cited before now showing up?
• Increased frequency: Are previously cited pages appearing more often?
• Query expansion: Are you being cited for a broader set of queries?

6.3 Iterate on the file

Based on the data, adjust:

• Swap pages: If a page never gets cited despite being in the top 10, replace it with one that has better answer‑potential.
• Refine annotations: If the model misinterprets a page, rewrite the annotation to be more explicit. For example, if you wrote “Guide to churn” and the model pulls your general churn page instead of the predictive-specific one, disambiguate the annotation.
• Reorder sections: Move your highest‑performing section to the top. Models weight early links more heavily.
• Update for new content: When you publish a major piece that targets an AI‑answerable query, add it to the file and consider removing a stale link.

Pro tip: Keep a changelog. Whenever you modify llms.txt, note the date and reason. This helps correlate changes with citation shifts.

Step 7: Scale Your llms.txt Strategy Across Content Properties

If you manage multiple domains or a large site with sub‑sections, you can scale the approach.

• Multi‑domain: Each domain gets its own llms.txt at the root. Cross‑link between them only if the domains are clearly related (e.g., a brand’s main site and its docs site). Otherwise, keep them separate.
• Sub‑sections: For massive sites, consider a hub‑and‑spoke model. The root llms.txt lists the most important pages and points to section‑specific llms.txt files. For example, https://acme.com/llms.txt links to https://acme.com/docs/llms.txt and https://acme.com/blog/llms.txt. This is outside the core spec but works in practice - some assistants will recursively fetch.
• Automate generation: If your site runs on a headless CMS or you have a content strategy powered by a data lake, you can script the generation of llms.txt from your content API. Just make sure the annotations are hand‑written or generated by a model that understands entity‑rich summaries.

Real‑World Expectations

Don’t expect an overnight explosion of citations. llms.txt improves retrieval when everything else is in place: answer‑engine optimized content, consistent brand mentions and backlinks, strong E‑E‑A‑T signals, and a clean crawl environment. It’s a multiplier, not a standalone solution.

I’ve seen sites go from zero AI citations to steady weekly appearances within two weeks of deploying a properly structured llms.txt alongside an AEO content brief workflow. But I’ve also seen sites see no movement because their underlying content wasn’t citable. The difference is always content quality.

If you want to test whether your content is citable, use a blind‑test approach to have models evaluate it. You’ll quickly see if your pages hold up.

Common Mistakes to Avoid

• Including marketing pages. Leave out your pricing page, “Request a Demo,” and gated content. AI assistants rarely cite these, and they waste your 30‑link budget.
• Using generic annotations. “Click here to learn more” tells the model nothing. Be specific.
• Neglecting mobile performance. If the linked pages are slow or not mobile‑friendly, AI crawlers may still use them, but real users who click through will bounce, hurting your E‑E‑A‑T signals long‑term.
• Setting and forgetting. llms.txt is not a one‑time task. Revisit it quarterly.

Key Takeaways

• llms.txt is a lightweight, Markdown‑formatted file that guides AI assistants to your most authoritative content.
• It works best when paired with answer‑engine‑optimized pages and a robust AI visibility tracking setup.
• Limit to 30 annotated URLs, grouped by user journey, with a concise blockquote summary at the top.
• Validate with manual AI testing and automated instruments; iterate based on citation data.
• Don’t expect miracles - llms.txt amplifies existing content quality, it doesn’t replace it.

If you’re starting from zero AI citations, this is one of the highest‑lever actions you can take in an afternoon. Ship it, track it, and watch how ChatGPT, Perplexity, Gemini, and Claude respond. Then come back and optimize.