AI Crawler Control: GPTBot, ClaudeBot and What robots.txt Really Does

AI Crawler Control: GPTBot, ClaudeBot and What robots.txt Really Does

Bottom line: Your AI crawler robots.txt rules decide whether ChatGPT, Claude, Perplexity, and Google's AI answers can quote you. The catch: training bots and retrieval bots are separate user agents. Block a training crawler and you lose nothing today. Block a retrieval crawler by accident and you silently disappear from AI answers, with no error to warn you.

We keep finding the same fixable mistake in technical audits: a founder pastes a "block AI bots" snippet from a forum, feels safer, and never realizes they just deleted their brand from ChatGPT Search. The snippet blocked the wrong crawler. Nothing broke, nothing errored, and traffic from AI answers just quietly stopped - the most expensive kind of SEO problem, because it looks like nothing at all.

Key Takeaways:

• —Every major AI company runs at least two crawlers - one for training (builds the model) and one for retrieval (fetches live content to cite in answers). robots.txt controls each independently.
• —Blocking a training crawler (GPTBot, ClaudeBot, Google-Extended) removes you from future model training but does not affect whether you get cited in live AI search.
• —Blocking a retrieval crawler (OAI-SearchBot, Claude-SearchBot, PerplexityBot) removes you from AI answers right now - usually by accident, via a copy-pasted Disallow block.
• —Ahrefs found GPTBot is the single most-blocked AI bot, yet only ~6% of all sites block it - most sites stay open, and many blockers don't realize search bots are a separate agent.
• —ChatGPT-User and other user-triggered fetchers may ignore robots.txt because a human, not a crawler, initiates the request - so blocking is not a privacy guarantee.

What is an AI crawler robots.txt rule, and why does it matter?

A robots.txt rule is a line in the plain-text file at yourdomain.com/robots.txt that tells a named bot which URLs it may or may not fetch. For decades this governed one thing: Googlebot and its search-engine cousins. That was simple because search engines had one job - index pages to rank them.

AI broke the simplicity. A modern AI company doesn't run one bot. It runs a fleet, and each bot has a different job. Some collect text to train the next model. Others fetch pages in real time to build the answer a user is reading in ChatGPT or Perplexity this second. They share a company name and nothing else. Your robots.txt treats them as separate strangers - which means one Disallow line can have wildly different consequences depending on which agent it names.

Cloudflare's network data shows GPTBot became one of the most active crawlers on the web, ahead of most traditional search bots by request volume. The bots are already at your door. The only question is which ones you're letting in - and whether you meant to.

Training crawlers vs retrieval crawlers: what's the actual difference?

This is the distinction that costs brands citations, so it's worth being precise.

A training crawler collects your content to help build or fine-tune a model. If GPTBot reads your page today, a fragment of that knowledge may surface in a future model version - months later, with no link back to you. Blocking it protects your content from being ingested for training. It has zero effect on whether you appear in AI answers today, because today's answers come from a different bot.

A retrieval crawler fetches content to answer a live query. When someone asks ChatGPT Search or Perplexity a question, a retrieval bot pulls candidate pages, and the engine cites the ones it uses - usually with a visible link. This is the traffic GEO practitioners actually chase. Block this bot and you remove yourself from the citation pool. No warning, no 404, no dashboard alert.

Here's the failure mode in one sentence: people block training crawlers to protect their content, grab a broader snippet to "be safe," and take out the retrieval crawler in the same block - deleting themselves from AI answers while thinking they only opted out of training. If you want the mechanics of how engines choose sources once they can reach you, we broke that down in how Google decides which pages to cite in AI Mode.

The per-crawler decision table (the one competitors leave out)

Most "block AI bots" guides hand you a wall of Disallow lines and never tell you what each one costs. That's the gap. Below is the table we actually use in audits - every major AI crawler, sorted by whether it's a training or retrieval agent, and the exact consequence of blocking it.

Read the "Block it and…" column twice. The bolded rows are the landmines. Blocking GPTBot is a defensible business choice - you're opting out of training. Blocking OAI-SearchBot, Claude-SearchBot, or PerplexityBot is almost never what someone intends, yet it's exactly what a broad User-agent: * block does. OpenAI documents this split plainly in its official bots reference: GPTBot trains, OAI-SearchBot surfaces you in search, and they obey robots.txt independently. Anthropic split its bots the same way - Search Engine Journal covered the more granular Claude bot controls - so ClaudeBot and Claude-SearchBot must be handled as separate decisions.

One more trap in that table: ChatGPT-User and Claude-User are user-triggered. OpenAI's documentation notes these fetches are initiated by a person, so robots.txt rules may not apply the way they do to automated crawlers. Blocking them is not a reliable privacy wall - treat it as a preference signal, not a lock.

How do I block AI training but keep AI search citations?

This is the configuration most growth-stage sites actually want: stay out of model training, stay inside AI answers. It's a selective allow-list, and it takes about five lines. Here's the process we run.

1. Pull your live robots.txt. Open yourdomain.com/robots.txt in a browser. Read every existing User-agent block - the damage is usually already there in a stale Disallow: / you forgot about.
2. List the training crawlers you want to block. For most brands that's GPTBot, ClaudeBot, and the Google-Extended token. These are training-only; blocking them costs you nothing in live AI search.
3. Explicitly allow the retrieval crawlers. Do not rely on silence. Add Allow: / blocks (or simply leave them unblocked and remove any wildcard that catches them) for OAI-SearchBot, Claude-SearchBot, and PerplexityBot.
4. Kill the catch-all landmine. If you have a User-agent: * with a broad Disallow, confirm it isn't silently swallowing your retrieval bots. Specific agent rules should override it, but many CMS-generated files get this wrong.
5. Validate, then wait. Test the file, then give engines time to re-read it. For search, expect roughly a day between a robots.txt change and it taking effect. Don't judge results the same afternoon.

A minimal version reads: block GPTBot, ClaudeBot, and Google-Extended; leave OAI-SearchBot, Claude-SearchBot, and PerplexityBot free to crawl. That's the whole trick - you're separating the training decision from the citation decision, which is the entire point of this article.

If you'd rather not eyeball raw directives, run the page through our AI SEO audit tool - it flags crawler-access conflicts and tells you which AI answer engines can currently reach the URL, so you catch a bad Disallow before it costs you a quarter of citations.

Should you block AI crawlers at all?

Blocking has a real cost, and the data says most sites overreact. Ahrefs found GPTBot is the single most-blocked AI bot, yet just 5.89% of all websites block it - and ClaudeBot's block rate, while still small, grew faster over the past year than any other bot's. Elite publishers block aggressively to protect licensable content - Originality.AI's live tracker shows the top 1,000 sites blocking at far higher rates than the web at large - but across the broad web, most sites stay open. For a growth-stage brand chasing visibility, open is usually correct.

The strategic question isn't "block or allow." It's "block which layer." A few honest scenarios:

• —You want AI visibility (most SaaS, DTC, and agencies): allow retrieval bots, optionally block training bots. You lose nothing in citations and keep your content out of model training if that matters to you.
• —You license or monetize content directly (major publishers, data providers): block training crawlers hard, and negotiate. Your content is the product.
• —You have sensitive or gated material: don't lean on robots.txt at all. It's a request, not a firewall. Use authentication.

Blocking training bots is often a wash for visibility, because retrieval - not training - is what puts you inside today's AI answers. If your goal is getting cited, the training decision barely moves the needle. The retrieval decision is everything. We unpack the broader mechanics in our Generative Engine Optimization guide, and if you're weighing a full opt-in strategy, our AI SEO service is built around exactly this trade-off.

Does robots.txt actually stop AI crawlers?

Partly, and the gaps matter. robots.txt is a voluntary standard - a polite request that well-behaved bots honor. The major named crawlers (GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot) do respect it, so for those, your rules hold.

Three things it does not do. First, it doesn't stop bots that ignore the standard - plenty of scrapers never check the file. Second, user-triggered fetchers like ChatGPT-User may bypass it because a human initiated the request. Third, blocking a crawler doesn't retroactively remove content already trained or already indexed; it only affects future access. And notably, blocking a crawler doesn't reliably stop citations either - engines can still reference a brand from other sources. robots.txt shapes access, not the whole information economy around your brand. If you need a hard boundary, that's a job for auth and server-level rules, not a text file at your root. llms.txt is a related emerging proposal - we covered whether llms.txt does anything yet - but it's not a substitute for getting your robots.txt right first.

How We Assessed This

The crawler classifications and block-consequences in this article were built from primary sources, not aggregated blog summaries. We cross-checked each user agent against its owner's official documentation - OpenAI's bots reference for GPTBot / OAI-SearchBot / ChatGPT-User, Anthropic's crawler documentation for the ClaudeBot family, and Google's crawler docs for the Google-Extended token - then confirmed the training-versus-retrieval split held consistently across all three vendors. The blocking-rate figures are cited from Ahrefs and Originality.AI's live tracker, with Cloudflare's network data for crawl-volume context. In our own retainer work auditing growth-stage sites, crawler-access misconfiguration is one of the most common and least-noticed AI-visibility problems we find - precisely because it produces no error state. We validate real crawler reachability with log analysis and access checks rather than trusting the robots.txt file to behave as written, then re-verify over a full optimization cycle because robots.txt changes take time to propagate and are easy to regress on the next CMS deploy.

Frequently Asked Questions

What is the difference between GPTBot and OAI-SearchBot?

GPTBot is OpenAI's training crawler - it collects content that may train future models. OAI-SearchBot is the retrieval crawler that indexes pages to cite inside ChatGPT Search. Blocking GPTBot keeps you out of training but leaves ChatGPT Search citations intact. Blocking OAI-SearchBot removes you from those live citations. They're controlled independently in robots.txt.

Does blocking ClaudeBot remove me from Claude's answers?

No. ClaudeBot is Anthropic's training crawler. Claude's live search answers are served by Claude-SearchBot and Claude-User, which are separate agents. Blocking ClaudeBot opts you out of training only. To stay in Claude's cited answers, leave Claude-SearchBot allowed.

Will blocking Google-Extended hurt my Google rankings or AI Overviews?

No. Google-Extended is a control token that only governs whether crawled content trains Gemini and Vertex AI. Disallowing it does not affect Google Search rankings or your eligibility for AI Overviews, which run on standard Googlebot access. Search Engine Land documented this when Google introduced the token.

How long does a robots.txt change take to work?

Expect roughly a day for search-focused crawlers to reflect a robots.txt change, sometimes longer. Don't measure results the same day you edit the file. Training crawlers may take even longer, since training runs are periodic, not continuous.

Can AI still cite me if I block its crawler?

Sometimes, yes. Engines can reference your brand using content from third-party pages, cached data, or prior indexing. Blocking a crawler reduces direct access but doesn't erase every path to a citation. robots.txt controls crawling, not the entire information ecosystem around your brand.

Is robots.txt enough to keep content private?

No. robots.txt is a voluntary request, not a security control. Non-compliant bots ignore it, and user-triggered fetchers may bypass it. For genuinely private content, use authentication and server-level access rules - never a text file at your domain root.

Get your crawler setup audited before it costs you citations

A single stray Disallow line can quietly remove your brand from ChatGPT, Claude, and Perplexity answers - and you won't see it in any ranking report. If you're not certain which AI crawlers can currently reach your site, that uncertainty is the problem.

Run your URL through our AI SEO audit tool for a fast read on crawler access, or book a strategy call and we'll map your training-versus-retrieval setup against your actual AI-visibility goals. For more on getting cited once your crawlers are configured right, browse the SEO Magics journal.