llms.txt + AI crawler robots.txt
Generate AI crawler rules without blocking the wrong bots.
Create an llms.txt draft, decide which AI crawlers to allow, and copy a robots.txt policy that separates AI search visibility from model-training opt-out.
Start with the safe middle path
Most SEO-focused sites should not blindly block everything. A practical first policy is to allow search and user-triggered crawlers, block training crawlers, and monitor logs after launch.
AI crawler user agents
Use these as a starter list, then verify critical entries against vendor docs.
| Token | Company | Purpose | Default |
|---|---|---|---|
| GPTBot | OpenAI | Training and improving OpenAI models from public web content. | block-training |
| OAI-SearchBot | OpenAI | Search and retrieval surfaces that can help content appear in AI answers. | allow-search |
| ChatGPT-User | OpenAI | User-triggered fetches when a person asks ChatGPT to access a page. | allow-user |
| ClaudeBot | Anthropic | Claude training crawler. | block-training |
| Claude-SearchBot | Anthropic | Search indexing crawler for Claude discovery and citations. | allow-search |
| Claude-User | Anthropic | User-initiated Claude fetches. | allow-user |
| PerplexityBot | Perplexity | Perplexity indexing crawler. | allow-search |
| Perplexity-User | Perplexity | User-triggered fetches for Perplexity answers. | allow-user |
| Google-Extended | Control token for Gemini and Vertex AI model improvement use. | block-training |
|
| Applebot-Extended | Apple | Control token for Apple AI training use. | block-training |
| CCBot | Common Crawl | Open web crawl data that can be used by AI companies and researchers. | block-training |
| Bytespider | ByteDance | ByteDance crawler commonly discussed in AI crawler block lists. | block-training |
| FacebookBot | Meta | Meta crawler used for web data collection and link previews depending on context. | block-training |
| Amazonbot | Amazon | Amazon crawler that can appear in AI crawler lists and bot management tools. | block-training |
| cohere-ai | Cohere | Crawler token associated with AI training and retrieval lists. | block-training |
Reference sources
- llms.txt proposalProposed markdown format for helping LLMs find concise site context.
- Google robots.txt guideRobots.txt manages crawler traffic and is not a security boundary.
- Google-Extended documentationGoogle-Extended is a robots.txt token, not a separate HTTP user agent.
- OpenAI crawlersOpenAI documents GPTBot, OAI-SearchBot, and user-triggered fetchers.
- Perplexity crawlersPerplexity documents crawler controls and update timing.
- Cloudflare AI Crawl ControlCloudflare positions AI crawler management as an active webmaster need.
- ai.robots.txtOpen-source list of AI-related crawler tokens and server configs.