How to Check Robots.txt & Sitemap Coverage in AmICited

If AI crawlers can’t read your site, none of your other optimization matters.

Quick Steps

• Open the Robots.txt & Sitemaps section under Audit → Agent Accessibility.
• Confirm major AI and search crawlers are allowed (not disallowed) in your robots.txt.
• Check that your sitemap is declared in robots.txt via a Sitemap: directive.
• Review how many URLs your sitemaps list, and compare that to your published page count.
• Fix any blocking rules or missing sitemap declarations, then re-check to confirm healthy status.

What is robots.txt, and why does it decide whether AI can see you?

Robots.txt is a plain-text file that sits at the root of your domain (yoursite.com/robots.txt) and tells crawlers which parts of your site they’re allowed to request. It predates the AI boom by decades; it was originally built so search-engine bots like Googlebot wouldn’t waste time crawling admin pages, staging environments, or duplicate content. But the same mechanism now governs a much higher-stakes decision: whether GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and the other AI crawlers that feed today’s answer engines are permitted to fetch your pages at all.

Every well-behaved crawler checks robots.txt before requesting anything else. The file is organized into blocks, each starting with a User-agent line naming a specific bot (or * for all bots), followed by Allow and Disallow rules that specify which paths that bot may or may not fetch. A single blanket rule like Disallow: / under User-agent: GPTBot is enough to shut that crawler out of your entire site, and it will not appear in ChatGPT’s citations no matter how good your content is. This is why crawlability (the general property of a site being technically reachable by bots) is treated as the foundation layer of AI search visibility rather than an afterthought.

Robots.txt also has a second job: it’s typically where crawlers look for a pointer to your sitemap, via a Sitemap: directive. An XML sitemap is a structured list of the URLs on your site that you want crawlers to know about, often including metadata like last-modified dates. Where robots.txt controls permission, the sitemap controls discovery: it’s the map that tells crawlers what exists so they don’t have to find every page purely by following links. A site can have perfectly permissive robots.txt rules and still be under-indexed by AI systems simply because its sitemap is missing, unreferenced, or incomplete.

These two files matter more for AI visibility than they ever did for traditional SEO. Search engines have crawled the web for 25+ years and built enormous link graphs that let Googlebot discover pages even without a sitemap. AI crawlers are newer, often more conservative about how much of a site they’ll fetch, and more likely to be blocked outright by robots.txt rules that were written with only Googlebot in mind. A rule that quietly excludes “unknown” or aggressive-looking bots can catch GPTBot or ClaudeBot as collateral damage. That’s the specific failure mode the Robots.txt & Sitemaps check exists to catch, and it’s why this is one of the first things any AI accessibility audit should verify, before you spend time on content, schema, or anything else.

Where to find it

It’s the Robots.txt & Sitemaps section of Audit → Agent Accessibility. Agent Accessibility is the part of AmICited’s audit that focuses specifically on whether AI systems can technically reach and parse your content, as distinct from the content-quality and citation-tracking parts of the product. Robots.txt & Sitemaps is typically the first check in that section, because everything else the audit measures assumes the crawlers can get in the door in the first place.

What it checks

As the section explains, it looks at “whether the important AI and search crawlers are allowed to read the site, whether sitemaps are declared in robots.txt, and how many URLs those sitemaps list in total.” In short:

• Crawler access: are the major AI and search bots permitted (not disallowed) in robots.txt? AmICited evaluates your actual robots.txt rules against the user-agent strings of the crawlers that matter for AI search visibility, so you see, bot by bot, whether it’s allowed, blocked, or simply not addressed by any rule (which usually falls back to allowed under the wildcard block).
• Sitemap declaration: is a sitemap referenced from robots.txt so crawlers can find it? A sitemap that exists but isn’t linked from robots.txt (and hasn’t been submitted elsewhere) is much less likely to be picked up automatically.
• Sitemap coverage: how many URLs your sitemaps list in total, giving you a quick sanity check on whether your published pages are actually being advertised to crawlers.

Together these three checks answer the two questions that determine whether an AI system can even start evaluating your content: is this crawler let in, and does it know what to fetch once it’s here. A brand can be doing everything right on the content side (clear answers, strong entity signals, well-structured pages) and still be invisible in AI answers simply because one of these two conditions fails silently in the background.

How to use it

1. Confirm crawlers are allowed. If an important AI bot is blocked, fix the robots.txt rule; this is the highest-priority issue on the page. Check each blocked crawler against your intent: a rule that blocks GPTBot was probably written on purpose if you don’t want your content used for OpenAI’s model training, but if it’s also silently blocking OAI-SearchBot or PerplexityBot (the crawlers that actually power live citations, as opposed to model training) you’re losing visibility for no strategic reason. It’s worth deciding deliberately which crawlers you want to allow AI bots to crawl your site rather than inheriting a default that blocks everything indiscriminately.
2. Declare your sitemap. Make sure robots.txt points to your sitemap so agents can discover all your pages. This is a single line, Sitemap: https://yoursite.com/sitemap.xml, but it’s one of the most commonly missing details on otherwise well-built sites, especially on sites where the sitemap was generated automatically by a CMS but never wired back into robots.txt.
3. Check the URL count. A sitemap listing far fewer URLs than you have published means pages aren’t being advertised to crawlers. This is usually a sign of a stale sitemap (generated once and never regenerated), a sitemap index that isn’t linking all of its child sitemaps, or a CMS plugin that’s silently excluding a content type; recently published blog posts and dynamically generated pages are common victims.
4. Re-check after edits and confirm the Crawler access and Sitemap URLs tiles in the readiness summary turn healthy.

Crawler access is the foundation: get it right first, because everything else assumes the bots can actually reach your content. Once access and discovery are both confirmed healthy, the rest of the Agent Accessibility audit (rendering, structured data, response times) starts to matter, because those checks only pay off if the crawler was allowed to reach the page in the first place.

It’s also worth remembering that robots.txt and sitemap coverage aren’t a one-time fix. New sections launch, CMS migrations regenerate sitemaps with different defaults, and CDN or WAF rules get added long after the original robots.txt was written, any of which can silently reintroduce a blocked crawler or an incomplete sitemap. Some teams pair this check with server-log analysis to confirm crawlers are actually visiting the pages they’re allowed to, and with a dedicated llms.txt file that gives AI systems a curated summary of your most important content alongside the raw sitemap. If you’re not sure where robots.txt and sitemap coverage fit relative to your other technical work, a broader AI visibility audit walks through the full set of accessibility checks in sequence, and AmICited’s own AI visibility tooling tracks whether fixing these issues actually translates into more citations over time, pairing the technical fix with the outcome it’s meant to produce, rather than treating “crawlers allowed” as the finish line. For teams managing this across dozens of client sites, the same access-and-sitemap checks are one of the recurring line items covered under AmICited for agencies , since a single misconfigured robots.txt rule tends to repeat across templates and needs to be caught early in onboarding, not after a client asks why their brand never shows up in ChatGPT.