Your robots.txt file has quietly become one of the most important levers you have over how AI models see your brand, and this guide from AmICited (FlowHunt's free AI brand-visibility tracker) walks you through exactly how to use it. We explain what robots.txt is and how AI crawlers read it, the critical difference between bots that train models on your content and bots that fetch your pages to cite you in answers, a rundown of the real user-agents you will meet (GPTBot, OAI-SearchBot, ChatGPT-User, Google-Extended, PerplexityBot, ClaudeBot and more), a realistic example file that blocks training while keeping citations open, how these choices affect your visibility in ChatGPT, Perplexity, Gemini and Claude, and the caveats every site owner should know before editing a single line.
The big idea is simple. Robots.txt is no longer a yes-or-no gate for search engines. It is now a policy document that lets you say, in effect, do not use my content to train your model, but you are welcome to read it and cite my brand when someone asks you a question. Getting that distinction right is the whole game.
What robots.txt is and how AI crawlers use it
Robots.txt is a plain text file that lives at the root of your domain, for example https://yoursite.com/robots.txt. It follows the Robots Exclusion Protocol, a decades-old convention that lets you tell automated visitors which parts of your site they may or may not request. Each rule block names a User-agent (the bot) and lists Allow and Disallow paths that apply to it.
AI companies have adopted this same file to declare their crawlers. When an AI crawler visits, it identifies itself with a specific user-agent string and checks your robots.txt before fetching pages. If you have disallowed that agent, a well-behaved bot backs off. The important shift is that the AI ecosystem now runs many different bots from the same company, each with a distinct job, so a single blanket rule is rarely what you actually want.
Training bots vs answering bots: the key decision
There are two very different reasons an AI system reads your site, and confusing them is the most common mistake site owners make.
Training bots collect content in bulk to help build and improve future models. Blocking them keeps your material out of the training corpus, but it does nothing for or against whether an AI cites you today. Answering and citation bots do the opposite job: they fetch or index your pages so the AI can surface, summarize and link to your brand when a user asks a live question. If you block those, you can vanish from AI answers even though your SEO is perfect.
So the decision is not block AI or allow AI. It is: block training, allow citation. Many brands want their expertise excluded from model training for licensing or control reasons, while still being the source an answer engine quotes and links. Robots.txt lets you express both preferences at once because the two jobs use separate user-agents.
The major AI crawlers and what each is for
Here are the real user-agents you are most likely to encounter, grouped by what they actually do. Get the names exact, because a typo means your rule silently does nothing.
- GPTBot (OpenAI): bulk crawler that gathers content for training future GPT models. This is a training bot.
- OAI-SearchBot (OpenAI): indexes pages for ChatGPT search results and citations. This is an answering bot.
- ChatGPT-User (OpenAI): fetches a page live when a ChatGPT user or action requires it. On-demand, answering side.
- Google-Extended (Google): a robots.txt token, not a separate crawler in your logs, that controls whether your content trains and grounds Gemini and Vertex AI. It is independent of Googlebot.
- Googlebot (Google): the classic search crawler. It also feeds Google AI Overviews and AI Mode through normal indexing, so blocking it costs you regular search too.
- PerplexityBot (Perplexity): builds Perplexity's index so your pages can be cited. Answering side.
- Perplexity-User (Perplexity): fetches a page on demand for a specific user question.
- ClaudeBot (Anthropic): collects public content for training Anthropic models. Training bot.
- Claude-User (Anthropic): retrieves a page when a Claude user asks a question that needs it.
- Claude-SearchBot (Anthropic): indexes content to improve Claude's search answers and citations.
- anthropic-ai and claude-web: older Anthropic tokens still worth listing for backward compatibility.
- Amazonbot (Amazon): powers Alexa and Amazon answer experiences.
- Applebot and Applebot-Extended (Apple): Applebot serves Siri and Spotlight; Applebot-Extended is a robots.txt token controlling whether Apple uses your content for AI training. The Extended token never appears in your access logs.
- Bytespider (ByteDance): training crawler for ByteDance models. Historically aggressive and inconsistent about honoring robots.txt.
- CCBot (Common Crawl): builds the open Common Crawl dataset that many other models train on.
- Meta-ExternalAgent and Meta-ExternalFetcher (Meta): the Agent crawls for Meta AI and Llama training and indexing; the Fetcher grabs a specific page when a Meta AI user references it.
- cohere-ai (Cohere), Diffbot, and Timpibot: additional data and knowledge-graph crawlers you may choose to name explicitly.
A realistic example robots.txt
The snippet below reflects a common goal: keep your content out of model training, but stay fully available to the bots that put your brand into AI answers and citations. Adjust it to your own policy, and remember that user-agent names are case-insensitive but must otherwise match exactly.
# Block model training crawlers User-agent: GPTBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: ClaudeBot Disallow: / User-agent: anthropic-ai Disallow: / User-agent: CCBot Disallow: / User-agent: Bytespider Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: Meta-ExternalAgent Disallow: / User-agent: cohere-ai Disallow: / # Allow answer and citation crawlers User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / User-agent: Claude-User Allow: / User-agent: Claude-SearchBot Allow: / # Keep normal search and its AI features working User-agent: Googlebot Allow: / Sitemap: https://yoursite.com/sitemap.xml
How this helps your business and your AI visibility
AI answer engines are becoming a primary discovery channel. When someone asks ChatGPT, Perplexity, Gemini or Claude for a recommendation, the brands that get named and linked win the click and the trust. A robots.txt that accidentally blocks citation bots quietly removes you from that conversation, while a deliberate one keeps you eligible to be cited even if you opt out of training.
Tuning robots.txt is only half the picture, though. You still need to know whether the changes are working: which engines actually mention you, for which prompts, and against which competitors. That is what AmICited's AI Brand Visibility Report measures, so you can connect a crawler policy to real citation outcomes rather than guessing.
It pairs naturally with two sibling tools. The llms.txt Generator helps you publish a clean, model-friendly map of your best content for the answer engines you just allowed, and the ChatGPT Query Fan-Out Generator shows the spread of real questions users ask, so you can see which pages need to stay crawlable to capture them.
Best practices and caveats
Robots.txt is a voluntary directive, not a security control. Reputable operators like OpenAI, Anthropic, Google, Perplexity and Apple honor it, but the file cannot technically enforce anything, and less scrupulous crawlers can ignore it entirely. Never rely on robots.txt to protect private data. Put anything sensitive behind authentication or firewall rules instead.
- Do not block the bots that drive your citations. Disallowing OAI-SearchBot, PerplexityBot, Claude-SearchBot or Googlebot to stop AI is the single most costly mistake, because it also cuts the answers that would have named you.
- Keep training and answering rules separate. A blanket Disallow for a whole company can catch its citation crawler too. Name each agent deliberately.
- Match user-agent strings exactly. A misspelling means the rule is ignored and the bot proceeds as if unrestricted.
- Blocking Googlebot removes you from ordinary Google search as well as AI Overviews. Treat it as an all-or-nothing decision.
- For pages you truly want out of AI results, robots.txt plus a noindex signal is stronger than robots.txt alone.
- Re-audit after every CMS migration or template change, since deployments frequently overwrite robots.txt.
- Verify with AmICited's AI Brand Visibility Report after each change so you can confirm you are still being cited where it matters.
Frequently asked questions
Does robots.txt actually stop AI companies from using my content?
It stops the ones that choose to comply, which includes every major operator such as OpenAI, Anthropic, Google, Perplexity, Apple and Amazon. They read your file and back off from paths you disallow. But robots.txt is a request, not a technical barrier, so a crawler that decides to ignore it can still fetch public pages. For anything you must keep private, use authentication or server-side blocking rather than robots.txt.
What is the difference between GPTBot and OAI-SearchBot?
GPTBot is OpenAI's training crawler. It collects content in bulk to help build future GPT models, so blocking it keeps your material out of that training data. OAI-SearchBot is the answering side. It indexes your pages so ChatGPT search can surface and cite your brand in live responses. Blocking GPTBot while allowing OAI-SearchBot is the classic setup: opt out of training but stay eligible to be cited.
Will blocking AI training bots hurt my AI visibility?
Not if you do it carefully. Training bots like GPTBot, ClaudeBot and CCBot only affect whether your content joins model training sets. They have no bearing on whether an answer engine cites you today. Your visibility depends on the answering and citation bots, such as OAI-SearchBot, PerplexityBot and Claude-SearchBot. As long as those remain allowed, you can block training and still be quoted and linked in AI answers.
Is Google-Extended the same as Googlebot?
No, and this trips up many site owners. Googlebot is the standard search crawler that indexes your pages, and that same index feeds AI Overviews and AI Mode, so blocking Googlebot damages both normal search and AI features. Google-Extended is a separate robots.txt token that only controls whether your content is used to train and ground Gemini and Vertex AI. You can disallow Google-Extended for training while leaving Googlebot fully allowed.
Why do Perplexity and Claude each have more than one user-agent?
Because they separate bulk crawling from live user requests. PerplexityBot builds Perplexity's search index, while Perplexity-User fetches a specific page when someone asks a question about it. Anthropic runs three: ClaudeBot for training, Claude-SearchBot for indexing citations, and Claude-User for on-demand retrieval. This granularity is helpful, because it lets you opt out of training while keeping the on-demand and citation bots free to bring your brand into answers.
How do I check whether my robots.txt is helping or hurting my AI presence?
Editing the file is only the first step. You need to confirm the outcome. AmICited's free AI Brand Visibility Report shows which engines mention your brand, for which prompts, and how you compare to competitors, so you can tie a crawler policy to real citations. Pair it with the llms.txt Generator to publish a clean content map and the ChatGPT Query Fan-Out Generator to see the questions your allowed bots should be able to answer with your pages.
Pairs well with
- AI Brand Visibility ReportDiscover how well AI assistants like ChatGPT, Claude, and Perplexity know your brand. Get visibility scores, sentiment analysis, and competitive insights across models — including who gets mentioned alongside you.
- ChatGPT Query Fan-Out GeneratorTransform any prompt into multiple keyword clusters optimized for AI search. Generate diverse search variations that help AI models like ChatGPT, Claude, and Perplexity better discover and recommend your content.
- LLMs.txt GeneratorTurn any site into a clean llms.txt — the curated, link-first summary AI assistants read instead of crawling every page. Give it a domain or a sitemap.xml URL.
- LLM Watermark DetectorInspect pasted text for deterministic watermark signals such as hidden Unicode characters. Results identify signals, not authorship or a proprietary watermark.
- LLM Watermark RemoverApply a best-effort rewrite and deterministic cleanup to pasted text. Review the result before using it; removal is not guaranteed.