Recommended default
Block this crawler when your first priority is opting out of AI training or broad data collection.
User-agent: Google-Extended Disallow: /
Decision notes
- Keep crawler-specific groups separate so future policy changes are easy.
- Do not use robots.txt for private data; use authentication or server-side access control.
- Re-check vendor documentation when this crawler becomes important to your site.
Source
https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers#google-extended