llms.txt and Agent Manifest Pages: A Maintained Machine-Facing Site Index
Build and maintain an llms.txt page that gives AI agents an accurate site index without exposing secrets, duplicating content, or drifting out of date.
An llms.txt and agent manifest page is a machine-facing index that tells AI systems what a site represents, which public pages are authoritative, and—when the business supports agent actions—which verified capabilities and policies apply. It is a map to maintained sources, not a substitute for those sources and not a promise that any particular crawler will use it.
Its contract is identity → scope → authoritative destinations → optional capabilities → constraints → freshness. The file succeeds when a machine can retrieve it, interpret it without guessing, follow live canonical URLs, and arrive at facts that still match production reality.
Questions it answers
The primary question is: “Which parts of this site should an AI system use to understand the organization, its content, and its supported actions?” Supporting questions include:
- What is the site’s canonical name, domain, purpose, and audience?
- Which product, service, documentation, pricing, policy, and support pages are authoritative?
- Which pages should be preferred over archives, campaign pages, parameters, or duplicate regional versions?
- Does the site expose an actual agent capability, or only human-readable information?
- Where are authentication, rate limits, data handling, commercial terms, and support documented?
- Which statements are descriptive guidance rather than access-control rules?
- Who owns the file, what event triggers an update, and how is drift detected?
When to use this post type
Use this type when the site has enough public, durable content to benefit from a curated machine-facing index and someone can own its maintenance. The reason to publish it is to reduce ambiguity for retrieval systems, not to create one more URL for its own sake.
| Confusable post type | What it organizes | Primary consumer | Choose it instead when |
|---|---|---|---|
| llms.txt and agent manifest page | Canonical identity, high-value public sources, and optional verified agent capabilities | AI retrievers, crawlers, agents, and the teams validating them | The deliverable is a concise machine-facing map at a predictable location |
| directory index | A collection of profiles, resources, locations, or listings | A person browsing and filtering a collection | Discovery paths, categories, descriptions, and human comparison are the main experience |
| documentation article | One product behavior, field, limit, configuration, or version | An existing user seeking an exact reference answer | The page must explain the destination content rather than merely point to it |
| policy page | Authoritative rules, obligations, scope, exceptions, and effective dates | People or systems deciding what is permitted | The policy itself must be read, accepted, or enforced; link to it from the manifest |
| agentic product data page | Products, identifiers, offers, availability, and transactional facts | Agents comparing or acting on product data | Item-level commercial data and actions are the main payload rather than a site-level index |
Do not confuse guidance with control. robots.txt expresses crawler access preferences; an XML sitemap helps crawlers discover URLs; authentication and authorization decide whether an action may occur. llms.txt provides curated context. A sentence in llms.txt cannot grant access, revoke access, protect a secret, or override a destination page’s terms.
Ready to Monitor Your AI Visibility?
Track how AI chatbots mention your brand across ChatGPT, Perplexity, and other platforms.
Best for these business types
- SaaS . The strongest fit because a software company usually has distinct product, feature, pricing, integration, API, security, status, and documentation sources. The index can resolve which page owns each fact, while a separate capability manifest can describe only actions the product genuinely supports.
- Ecommerce . Strong when products, shipping, returns, availability, and customer-service policies are public and canonical. Keep volatile item data in feeds or APIs; use the index to point to those maintained sources rather than copying a catalog into Markdown.
- Marketplaces . Valuable when buyer, seller, provider, and platform policies differ. Label each audience and jurisdiction so an agent does not apply seller rules to a buyer or infer platform inventory from one listing.
- B2B services . Useful for clarifying capabilities, industries, service boundaries, evidence, procurement material, and contact routes. Do not convert a negotiated scope or client-specific promise into a universal machine-facing claim.
- Agencies . Useful when the firm maintains many service, methodology, case-study, and expertise pages. Client portals, credentials, private reports, and internal playbooks stay out of the public file.
- Manufacturers and industrial businesses . Useful for directing systems to product families, specifications, certifications, manuals, distributors, and safety documents. The index must never paraphrase safety-critical instructions when the controlled document is the authority.
Small brochure sites with five stable pages may gain little from another maintained artifact. Sites without a clear content owner should fix canonicalization, navigation, and source quality before publishing a file that will immediately drift.
Search intent
search intent
is the result expected from a query. This type has two audiences with different intent. A machine fetches a predictable root path and expects concise Markdown, stable headings, canonical links, and no decorative noise. A human searcher usually wants implementation guidance: “llms.txt example,” “what belongs in llms.txt,” or “agent manifest format.” The public explanatory page can answer those questions, while the deployed /llms.txt remains optimized for machine retrieval.
The file itself is not a keyword landing page. Do not add generic definitions, repeated category terms, or hundreds of blog links to make it “rank.” Every extra line consumes attention and creates another maintenance obligation. Prefer ten deliberate links with clear descriptions over a dump of ten thousand URLs.
Because conventions and consumer support can change, state what your implementation is based on and avoid claiming universal adoption. A successful fetch proves only that the file is accessible and parseable; it does not prove that a particular AI product uses it for ranking, retrieval, training, or citation.
Stay Updated on AI Visibility Trends
Get the latest insights on AI mentions, brand monitoring, and optimization strategies.
Page structure
Word bands are editing constraints, not targets. The deployed file should stay concise enough to audit line by line. The human-facing implementation note can be longer, but it must not be copied into the machine file.
| Section | Word band | Purpose | Required? |
|---|---|---|---|
| Site name and direct description | 30–70 | Establish canonical identity, purpose, audience, and scope before any links. | Required |
| Scope and interpretation note | 30–90 | Explain what the index covers and point to controlling access or policy sources. | Required when ambiguity is likely |
| Primary resources | 60–180 | Link to the small set of pages that define the organization, offering, documentation, pricing, and support. | Required |
| Topic or product groups | 80–300 | Organize additional canonical resources under plain, stable headings. | Conditional; use only when the catalog warrants it |
| Agent capabilities | 80–250 | Identify actual actions and link to their machine-readable contracts, authentication, limits, and policies. | Conditional; omit when no supported action exists |
| Optional resources | 40–150 | List useful but nonessential material such as research or selected case studies. | Conditional |
| Maintenance record | 20–70 | State verification date, owner role, source system, or generated status. | Required |
A typical curated file is roughly 200–700 words. Length is not a quality signal: the right size is the smallest index that establishes identity and routes a consumer to maintained sources without hiding key distinctions.
Required elements
The index must be boring in the best sense: predictable, explicit, and easy to diff. Put critical interpretation before optional links so a partial read does not produce a false conclusion.
| Element | Always or conditional | Position | Production rule |
|---|---|---|---|
| direct answer block | Always | First lines after the H1 | Name the organization and state what the site provides in language that stands alone. |
| quick overview and table of contents | Conditional | After the description | Use simple Markdown headings as navigation when several resource groups exist; do not add a decorative web TOC to the raw file. |
| spec table | Conditional | Human-facing implementation page | Document endpoint, format, owner, generation source, validation, and refresh triggers; avoid HTML tables in the raw file. |
| note box | Conditional | Beside interpretation guidance | Clarify that indexing guidance does not replace permissions, policies, or destination-page facts. |
| warning box | Conditional; mandatory for exposure risk | Before capability or private-data guidance | Name the risk and safe source; never place secrets, tokens, nonpublic endpoints, or customer data in a public manifest. |
| sources block | Always on the implementation page | After the specification | Name the convention, internal source-of-truth systems, and validation evidence without implying unsupported standardization. |
| freshness stamp | Always | End of the raw index or near the top of the implementation record | State the last substantive verification and owning role. |
| update log | Conditional | Human-facing implementation page | Record changes to scope, important destinations, capabilities, or generation rules—not punctuation edits. |
| related content block | Always on the implementation page | Before FAQ | Link to access controls, structured data, product data, and measurement guidance with a reason for each. |
| FAQ structure | Always on the implementation page | Before CTA | Answer residual questions about adoption, scope, security, duplication, and maintenance. |
| CTA block | Always on the implementation page | Final element | Offer a validation, monitoring, or implementation action appropriate to a consideration-stage reader. |
Frontmatter
Follow the frontmatter specification
. On this post-type specification, use entity = "post-type-llms-txt-page" and schemaType = "Article". On a human-facing implementation page for a specific organization, use a stable identity such as acme-ai-access-index, not a campaign phrase or date.
Use Article because the web page explains the implementation. schema markup
describes visible content; it does not turn the raw text file into a recognized agent protocol. Do not mark the page as SoftwareApplication, Dataset, or HowTo unless its visible content and template satisfy the relevant requirements independently.
The raw /llms.txt file normally has no frontmatter because frontmatter must not leak into the published output. Store its operational metadata in the CMS, generator configuration, or repository record: canonical domain, locale scope, owner, source collection, generation mode, last verified date, next review rule, validator result, and alert destination. If localized files exist, document the selection rule and keep one unambiguous canonical root response.
Full example
This fictional file demonstrates a concise index for a SaaS platform. Its URLs, product, and capability are examples; the pattern is the specification.
# Northstar Analytics
> Northstar Analytics is a reporting platform for operations teams. This index points to the public pages that define the product, plans, documentation, policies, and supported agent capability.
Access permissions are controlled by robots.txt, authentication, and the policies linked below. This file does not grant access or permission to reuse content.
## Product
- [Product overview](https://www.northstar.example/product): Current product scope and supported reporting workflows.
- [Plans and pricing](https://www.northstar.example/pricing): Current public plans, included features, and billing terms.
- [Integrations](https://www.northstar.example/integrations): Supported data sources and destination systems.
## Documentation
- [Documentation home](https://docs.northstar.example/): Current user and administrator documentation.
- [API reference](https://docs.northstar.example/api/): Public endpoints, schemas, authentication, errors, and rate limits.
- [Release notes](https://docs.northstar.example/releases/): Dated changes to product and API behavior.
## Trust and support
- [Security](https://www.northstar.example/security): Security program and current assurance documents.
- [Privacy policy](https://www.northstar.example/privacy): Data processing, retention, and user rights.
- [Support](https://www.northstar.example/support): Supported contact routes and service-status link.
## Agent capability
- [Report export action](https://docs.northstar.example/agents/export-report): Authenticated action contract, accepted inputs, output format, rate limits, and error handling. Availability depends on the user's plan and role.
## Optional
- [Research library](https://www.northstar.example/research): Original benchmark reports with methods and publication dates.
Verified 2026-08-27 by the Documentation Operations team. Generated from the canonical public-resource registry; validate after product, plan, policy, API, or URL changes.
The example declares one capability only because a real, documented action contract exists. If the product has no supported agent action, omit that section. Never infer a transactional capability from the presence of a search box, form, or undocumented endpoint.
For an agent manifest stored separately from /llms.txt, keep the same discipline. Specify a versioned format, canonical identifier, production endpoint, authentication method, allowed operations, input and output schema, rate limits, consent boundaries, error states, and policy URLs. Validate it against the live system. A syntactically valid declaration that advertises a disabled action is still wrong.
Design gallery
The raw file has little visual design by intent. Gallery variants should test information architecture, scan order, mobile readability of the human implementation page, and operational evidence—not decoration.
Quality checklist
Publish only when every applicable statement is true:
- The file resolves at the intended root URL without authentication, redirect loops, consent walls, or a rendered application shell.
- The response is readable plain text or Markdown, uses UTF-8, and does not depend on JavaScript to reveal its content.
- The H1 gives the canonical organization or site name, and the description states purpose, audience, and scope without slogans.
- Every linked URL is canonical, public, indexable by policy, reachable, and owned by the organization or clearly labeled as external.
- Link descriptions say what authority the destination holds; they do not repeat generic anchor text such as “learn more.”
- Primary product, pricing, documentation, policy, and support sources agree with the index.
- Archive pages, search results, tracking parameters, duplicate locales, campaign pages, and low-value tag pages are excluded.
- Capability claims match a live, supported, authenticated contract and include the relevant constraints.
- No secret, token, private endpoint, personal data, client document, unpublished roadmap item, or security-sensitive implementation detail appears.
- Access, permission, licensing, and policy language points to controlling sources and is not contradicted by the index.
- The file does not claim guaranteed ranking, citation, training exclusion, or universal consumer support.
- Locale and regional scope are explicit wherever pricing, policies, availability, or documentation differ.
- The owner, source registry, generation process, and validation method are recorded outside or at the end of the file.
- Broken-link, unexpected-redirect, response-status, content-hash, and required-section checks run after relevant deployments.
- The verification date changes only after destinations, descriptions, capabilities, and policies are substantively checked.
Common mistakes
Treating the file as a sitemap. A complete URL inventory destroys prioritization and is hard to review. Keep XML sitemaps for discovery; curate llms.txt around authoritative sources and meaningful groups.
Treating it as access control. A request in Markdown is not an enforcement layer. Express crawl rules in robots.txt, protect private resources with authentication, and place binding requirements in the relevant policy and system controls.
Copying destination content into the index. Repeated pricing, product specifications, and policies diverge. Summarize only enough to identify authority, then link to the maintained source.
Publishing speculative capabilities. An undocumented form or API route does not make a site agent-ready. Declare only production-supported actions with authentication, schemas, constraints, errors, and an owner.
Including everything “just in case.” More links create more ambiguity and more failure points. Optional content should earn inclusion by answering a likely retrieval need that primary sections do not cover.
Exposing private material. Public machine-readable files are public. Never list staging hosts, internal APIs, credentials, customer exports, unpublished documents, or security details that were not intentionally approved for publication.
Generating without governance. Automation can reproduce bad source data at speed. A generator needs an approved source registry, exclusion rules, deterministic ordering, validation, review ownership, and deployment alerts.
Hand-editing a generated file. The next generation overwrites the fix. Correct the source record or generator, regenerate, and record the material change.
Claiming unsupported outcomes. “This guarantees AI citations” turns an uncertain implementation convention into a misleading promise. Report accessibility and retrieval evidence separately from visibility and citation outcomes.
Updating the date without checking reality. A new timestamp cannot repair a dead documentation URL, renamed plan, or disabled action. Verification means comparing every important declaration with its production source.
Internal linking
Link the human implementation page upward to SEO post types when an author must distinguish the machine index from a directory, documentation page, or policy. Link from each operational claim to its controlling source: product scope to the product page, current prices to pricing, behavior to documentation, permissions to access controls, and obligations to policy.
The raw file should use canonical absolute URLs because it may be retrieved outside normal site navigation. Prefer one authoritative destination per fact. If two pages overlap, resolve ownership before listing both; the index should expose the source hierarchy, not memorialize internal disagreement.
Use short, stable section names such as Product, Documentation, Policies, and Agent capabilities. Keep locale variants in clearly labeled groups only when they differ materially. Do not link every blog post; select durable research or guides only when they help a system understand the site’s subject and evidence.
Inbound links matter operationally too. Documentation, developer portals, and AI-accessibility guidance should point maintainers to the implementation record, while the record points to the live file and validator. That creates a review path for humans without cluttering the machine index.
How to measure results
Measure the file as infrastructure first and as a visibility input second. A citation increase cannot be attributed to llms.txt merely because both happened after publication.
Track four layers:
- Availability: root URL response status, redirect behavior, content type, encoding, latency, render independence, and uptime.
- Integrity: parse success, required sections, duplicate URLs, broken links, redirect targets, canonical mismatch, unauthorized domains, exposed secrets, and content hash changes.
- Freshness: days since substantive verification, destination changes since verification, source-registry coverage, owner acknowledgement, and time to repair drift.
- Outcomes: server-log fetches by identifiable agents where legally and technically appropriate, visits to indexed destinations, AI citations of preferred canonical pages, and answer accuracy for tracked brand or product questions.
Establish a baseline before deployment: which URLs are cited, which facts are misstated, whether the root file exists, and which crawlers request it. Annotate publication and every material update. Compare observation windows that are long enough to avoid reading one fetch or citation as a trend, and retain the distinction between correlation and causation.
Test the failure modes directly. Rename a staging copy of a listed URL and confirm the validator catches the break. Change a canonical mapping and confirm the generator updates the index. Disable a capability in a controlled test environment and confirm the manifest check fails. These tests prove the maintenance system, not external adoption.
Use how we measure results to separate technical availability, machine representation, discovery, citation, engagement, and business outcomes. In AmICited, review the live file under Agent Accessibility and use the Cockpit report to observe cited URLs and AI visibility alongside deployment annotations. The credible success claim is “the file is valid, current, and routes systems to the intended sources”; any downstream visibility change requires separate evidence.
FAQ
Frequently asked questions
What is an llms.txt page?
Is llms.txt the same as robots.txt or an XML sitemap?
Does publishing llms.txt improve rankings or guarantee AI citations?
What belongs in an agent manifest?
Should every site publish an llms-full.txt file?
How often should llms.txt be reviewed?
More tutorials in this section
Ready to put it into practice?
Free check · 7-day trial · no credit card