Technical guide · 30 August 2026
Classify OAI-SearchBot, GPTBot and PerplexityBot correctly
Blanket rules for AI crawlers often mix up two different decisions: should a page be discoverable in current AI search, and may its content be used for future model training? Each user agent needs to be assessed according to its documented purpose.
The short answer
OAI-SearchBot is relevant to ChatGPT search; GPTBot is not. GPTBot concerns OpenAI's potential use of content for model training. PerplexityBot supports search and linking in Perplexity. Google's AI Overviews and AI Mode rely on normal Google Search and therefore Googlebot; Google-Extended is not a separate search crawler and, according to Google, has no effect on Google Search.
Crawler, purpose and consequence
| User agent | Documented purpose | Control | What blocking means |
|---|---|---|---|
| Googlebot | Google Search, including the search foundation for AI Overviews and AI Mode | robots.txt and robots meta tags | The URL can lose crawl and Search eligibility, removing the foundation for Google's generative search features. |
| Bingbot | Bing's index, which also supports selected Copilot and grounding experiences | robots.txt and Bing-compatible robots meta tags | Content may not be indexed reliably or selected for supported grounding results. |
| OAI-SearchBot | Discovery and linking in ChatGPT search | robots.txt | OpenAI cannot fetch page content through this search crawler; under some conditions the bare URL may still be known. |
| GPTBot | Potential use of web content to improve future models | robots.txt | Training use is declined; this is a separate decision from ChatGPT search. |
| PerplexityBot | Discovery and linking in Perplexity search | robots.txt and, where relevant, network allow-lists | The page is unavailable to this search crawler; that says nothing about other systems. |
| Google-Extended | A control token for Gemini Apps and Vertex AI; not a separate HTTP user agent | robots.txt as a user-agent token | According to Google, it has no effect on inclusion, ranking or presentation in Google Search. |
A credible policy begins with the business decision
Before changing a file, establish which content should be publicly discoverable and which training uses the organisation accepts. The policy can then remain minimal and auditable.
- 1
Identify public search content
Only pages intended to be public and citeable for people belong in the search strategy.
- 2
Separate search from training
OAI-SearchBot and PerplexityBot are search crawlers; GPTBot represents a separate training decision.
- 3
Keep specific groups complete
Once a user agent has its own group, do not assume generic rules are automatically inherited.
- 4
Validate configuration, not imagined behaviour
A validator can test rules and reachability, but cannot claim that an external bot actually indexed or cited the page.
Example: keep a public website open to search crawlers
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xmlExample: allow ChatGPT search while blocking GPTBot separately
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xmlThe second example is not a blanket recommendation. It only demonstrates that search access and training access can be expressed separately.
What llms.txt does not do
llms.txt is a community proposal for a compact content index. It replaces neither robots.txt nor sitemaps, indexing, internal links or accessible HTML. Google explicitly says that its AI features require no special AI files or markup. llms.txt should therefore be treated, at most, as supplementary documentation — not a ranking lever.
What can be validated technically
- robots.txt is reachable and can be parsed
- Googlebot, Bingbot, OAI-SearchBot and PerplexityBot are not accidentally blocked from target paths
- the sitemap and canonical pages are reachable
- robots meta tags do not contradict crawl policy
- important content is present in delivered HTML
- the server or WAF does not return unintended errors to legitimate crawlers
Primary sources
From policy to a concrete website review
The AI crawlability review examines more than the text file: status codes, redirects, renderability and the actual target pages are part of the assessment.