robots.txt for AI crawlers: which agents to allow

AI crawlers split into two jobs, and the difference matters when you decide what to allow. Search and citation crawlers fetch pages so an engine can answer a question and link to you. Training crawlers collect text to train a model. Blocking the first removes you from AI answers. Blocking the second does not.

The agents worth naming

• OAI-SearchBot is OpenAI's search crawler. It is the one that decides whether ChatGPT can surface your page.
• ChatGPT-User fetches a page because a person in ChatGPT asked for it.
• GPTBot is OpenAI's training crawler. Separate decision from the two above.
• Claude-SearchBot and Claude-User are Anthropic's search and on-demand fetchers. ClaudeBot is the training crawler.
• PerplexityBot indexes pages for Perplexity's answers. Perplexity-User fetches on a user's request.
• Google-Extended is not a crawler. It is a flag that controls whether content Google already crawled can be used for Gemini. Disallowing it does not remove you from Google Search.

A robots.txt that keeps you citable

User-agent: *
Allow: /
Disallow: /api/
Disallow: /account

Sitemap: https://yoursite.com/sitemap.xml

A wildcard Allow: / already covers every agent above. You only need to name an agent when you want to treat it differently from the rest, for example allowing search crawlers while disallowing a training crawler:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

How to check what is actually happening

1. Read your own file first: curl -s https://yoursite.com/robots.txt. Look for a blanket Disallow: / left over from a staging setup.
2. Request your homepage as one of the agents and read the status code:
   curl -sI -A "OAI-SearchBot" https://yoursite.com
3. Repeat for Claude-SearchBot and PerplexityBot. A 200 is fine. A 403 means something refused the request.
4. If you get a 403 while a normal browser gets a 200, robots.txt is not the cause. Check the layer in front of your app next.

The block nobody chose

Most AI crawler blocks we see are not in robots.txt at all. They come from a CDN or firewall bot-protection rule that treats an unfamiliar agent as an attacker and answers 403. The site owner never decided to block anything. In Cloudflare this lives under bot management and managed rules; other providers have an equivalent. If the agent gets a 403 and robots.txt is clean, that is where to look.

Not sure which of the two is happening on your site? Run the free scan and it will tell you which layer refused, per engine.