AI crawler guide

Perplexity Robots.txt Rules

PerplexityBot and Perplexity-User robots.txt examples for AI answer visibility and crawl control.

Search intent

Users want to know whether blocking Perplexity affects AI answer visibility.

Practical checklist

  • Perplexity documents separate crawler controls and says changes may take time to reflect.
  • PerplexityBot is the indexing bot most sites will configure.
  • Perplexity-User is user-triggered and should be considered separately.

Copy-paste starter pattern

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: FacebookBot
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: cohere-ai
Disallow: /

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Related crawler pages