Crawler Policy
What OAI-SearchBot, GPTBot, and ChatGPT-User actually do
A plain-language breakdown of the OpenAI crawler distinctions that matter for publishers and B2B companies deciding what to allow.
Quick take
- Search access and training access are not the same decision.
- Publishers should decide crawler policy from outcome, not from fear or guesswork.
- Live verification matters because upstream bot controls can override repo settings.
These bots do different jobs
GPTBot is the training-related bot most publishers already know. OAI-SearchBot is about search discovery and summaries. ChatGPT-User represents requests initiated on behalf of users. Treating them as one bucket makes policy decisions sloppy.
Decide from business outcome
A publisher may want citation and referral traffic from ChatGPT search while still drawing a line around training. That is a different position from blocking everything. The policy should reflect the actual business goal, not a copied default.
The live file is what counts
If Cloudflare or another upstream layer is prepending a managed robots block, it does not matter what the repo says. Always verify the public file after changes so you know what OpenAI actually sees.
Sources
Related
About the author
Grail Research Team
Operators studying AI workflows, internal systems
The Grail Research Team writes about AI employees, workflow design, governance, and AI-search visibility with a bias toward operator reality over vendor theater. Learn more about Grail.