Recommended default
Block this crawler when your first priority is opting out of AI training or broad data collection.
User-agent: Bytespider Disallow: /
Decision notes
- Keep crawler-specific groups separate so future policy changes are easy.
- Do not use robots.txt for private data; use authentication or server-side access control.
- Re-check vendor documentation when this crawler becomes important to your site.