Strangers in your server logs
Open your server logs or your CDN dashboard and you will find visitors nobody introduced you to. GPTBot. ClaudeBot. OAI-SearchBot. CCBot. Bytespider. More than a dozen user-agents now crawl websites on behalf of AI companies, and the number keeps growing.
Sooner or later, someone on the management team asks the obvious question: should we block them?
It is one of the questions Mittelstand clients raise most often with us. Our honest answer is that “block or allow” is the wrong framing. These bots do very different jobs. Some collect text to train future models. Others fetch your page at the exact moment a buyer asks ChatGPT or Perplexity for a supplier recommendation. Treat them all the same and you either give away more than you intended, or you quietly remove your company from the answers your customers are already reading.
The good news: once you know which bot does what, the decision takes about ten minutes. Here is how we approach it for our own sites and for our clients.
Not all AI crawlers are the same: the four families
Before you touch your robots.txt, sort the bots by what they actually do. The vendors document this themselves, and the differences matter.
| Family | Main user-agents | What it does | What blocking it costs you |
| Training crawlers | GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Bytespider | Collects content to train future models | No documented effect on whether you appear in AI answers today |
| Live search crawlers | OAI-SearchBot, Claude-SearchBot, PerplexityBot | Builds the index an assistant searches when it needs a source for an answer | You drop out of that index, and with it out of citations |
| User-triggered agents | ChatGPT-User, Claude-User, Perplexity-User | Opens a page because a person just asked the assistant to | You turn away a real visitor who requested your page |
| Generic corpus crawlers | CCBot (Common Crawl), Amazonbot | Feeds large datasets that third parties reuse | You limit reuse by companies you never chose |
One detail surprises many marketing teams. Google-Extended only controls whether Google may use your content for Gemini training and grounding. It does not touch Google Search, and it does not remove you from AI Overviews, which rely on the regular Googlebot. Blocking it will not hurt your rankings, but it will not make you disappear from Google’s AI features either.
Training vs. live search: the split almost nobody uses
OpenAI and Anthropic both publish separate tokens for training and for search. That split exists for one reason: so a publisher can say “don’t train on my content, but do cite me.”
In practice, very few sites take advantage of it. A recent review of 36 robots.txt files by the French agency found that most sites in their sample never mention a single AI crawler. Not one site blocked a live answer engine, and only one used the training/search split deliberately.
Open the robots.txt of a typical Mittelstand website and you will likely see the same picture. Most robots.txt files were written years ago for Googlebot and have not been opened since. Nobody decided to let AI crawlers in or keep them out. The question simply never came up.
That matters, because silence is still a decision. If your file says nothing, every AI bot that respects robots.txt treats your whole site as open. For most B2B companies that is actually fine. But it should be a choice, not an accident.
The configuration we recommend
The useful question is not “AI: yes or no?” It is “which job do I want to allow?” Here is the logic we apply.
Live search crawlers: let them in. This is the only group where blocking has a clear, documented cost. Shut out OAI-SearchBot, Claude-SearchBot or PerplexityBot and those assistants can no longer use your pages as a source. For a company investing in GEO, that defeats the purpose.
User-triggered agents: let them in. Behind every ChatGPT-User or Perplexity-User request is a person who asked to see your page. Blocking them is like locking the showroom door while a customer is standing outside.
Training crawlers: your call. If your content is the product (paid research, a publishing business, proprietary data), blocking training bots is reasonable. If your content exists to market your products and services, the case for blocking is much weaker. Being part of what models learn about your category can help, not hurt.
Generic corpus crawlers: your call, with a lean towards blocking. CCBot feeds Common Crawl, which many third parties reuse for training. You have no say in who those parties are. If you want to restrict anything, this is the most common place to start.
A balanced robots.txt for a B2B company that wants citations but limits third-party reuse could look like this:
# Live AI search: allowed
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /admin/
Disallow: /checkout/
# User-triggered agents: allowed
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
Disallow: /admin/
Disallow: /checkout/
# Model training: decide per business (allowed here)
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /admin/
Disallow: /checkout/
# Third-party corpus: blocked
User-agent: CCBot
Disallow: /
# Everyone else, including Googlebot
User-agent: *
Disallow: /admin/
Disallow: /checkout/
Sitemap: https://www.example.com/sitemap.xml
One rule of the robots.txt standard is worth knowing here. A crawler follows only the most specific group that names it. As soon as you write a group for GPTBot, it ignores everything under User-agent: *. So if you exclude /admin/ for all bots, repeat that line in every named group too, or the named bots will crawl it.
Four mistakes that cost visibility
- Blocking AI through User-agent: *. A blanket Disallow: / meant to stop AI bots also stops Googlebot and Bingbot. It happens more often than you would think after a relaunch, when a staging rule goes live by mistake. Always target AI crawlers by name.
- Blocking the wrong family. Teams that want to “protect content from AI” often block PerplexityBot or OAI-SearchBot first, because those names are the most familiar. Those are precisely the bots that send citations and traffic back to you.
- Expecting robots.txt to undo the past. A new rule applies to future crawls. It does not delete anything a crawler collected last year.
- Confusing robots.txt with noindex.txt controls crawling, not indexing. A blocked URL can still appear in results if other sites link to it. If a page must not show up anywhere, use a noindex tag or proper access control.
Check your own site in five minutes
This is the quickest check in any technical SEO audit, and still one of the most skipped.
- Open https://your-domain.com/robots.txt in a browser, or run curl -s https://your-domain.com/robots.txt in a terminal.
- Search the file for the tokens listed in the table above: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, CCBot.
- For each one, note whether it is named, allowed, blocked or slowed with Crawl-delay. No mention means it follows your User-agent: *
- Make sure no live search crawler or user agent is blocked unless you truly intend it.
- If you get a 403 instead of a file, check your firewall or CDN. Some security settings block access to robots.txt itself, which leaves crawlers guessing.
Keep in mind that robots.txt states your intention. It does not prove that a bot obeys it. To verify real behaviour, you need server logs and a check against the IP ranges the vendors publish.
Frequently asked questions
Should I block GPTBot?
Only if you do not want OpenAI to use your content for training. Blocking GPTBot does not remove you from ChatGPT search results. That job belongs to OAI-SearchBot, which you can allow separately.
What is the difference between GPTBot and OAI-SearchBot?
GPTBot gathers content to train OpenAI’s models. OAI-SearchBot builds the search index ChatGPT uses to find and cite sources when it answers a question. You can block one and allow the other.
Does blocking ClaudeBot stop Claude from citing my website?
No. ClaudeBot is Anthropic’s training crawler. Claude-SearchBot handles search, and Claude-User fetches pages that users ask for. Each can be controlled on its own.
Is blocking CCBot enough to keep my content out of AI training?
Not on its own. CCBot feeds Common Crawl, which many third parties use. But the major AI companies also run their own training crawlers with their own names, so you would need to block those too.
Will blocking Google-Extended affect my Google rankings?
No. Google-Extended is a control token for Gemini training and grounding only. Google Search and AI Overviews use the standard Googlebot.
Do AI crawlers actually respect robots.txt?
The major vendors state that their crawlers do. Robots.txt is still a voluntary standard, though, so the only way to confirm compliance is to check your server logs against the IP ranges each vendor publishes.
The bottom line
Let in the bots that cite you. Think carefully about the ones that train on you. And open your robots.txt at least once a year to check that it still says what you think it says.
For most B2B and Mittelstand companies, AI search is becoming a real source of qualified enquiries. A line written years ago, or never written at all, should not decide whether you are part of that conversation.
At Matrixe ZoneMatrixe Zone, robots.txt is the first stop in every GEO and AI visibility audit we run. If you want to know how your brand currently shows up in ChatGPT, Perplexity, Claude and Gemini, and what is holding it back, talk to our team.
