Google-Extended
Google · Training crawlers · User agent only
About this crawler
A robots.txt token that controls whether Google may use crawled content for Gemini training and grounding.
Usually means: Public content is being collected that may end up in model training data or large web datasets.
Google-Extended has no HTTP user agent of its own: crawling is done with existing Google user agents, so it never appears in requests.
User agent
Google-Extended
Google-Extended is a robots.txt product token: Google reads rules addressed to it, but its requests arrive under another user agent.
IP verification
Google publishes no IP ranges for Google-Extended, so requests are identified by user agent only and recorded as unverified. Anyone can send this user agent.
Questions
- What is Google-Extended?
- A robots.txt token that controls whether Google may use crawled content for Gemini training and grounding. MetricRaven groups it under Training crawlers: crawlers collecting public pages that may be used to train AI models.
- How do I verify that a request really comes from Google-Extended?
- Google publishes no IP ranges for Google-Extended, so only the user agent identifies it. Anyone can send that string, so treat it as a claim.
- How do I block or allow Google-Extended?
- Add a robots.txt group for "Google-Extended". It has no HTTP user agent of its own, so it can't be blocked at the server.
- How can I see when Google-Extended visits my website?
- Most crawlers never run JavaScript, so browser analytics miss them. MetricRaven's @metricraven/ai-crawl package reports crawler requests from your server, verifies their IPs against published ranges, and shows which pages each crawler reads.
See when AI assistants and crawlers read your site
Add one line to your server and MetricRaven shows which assistants, search engines and training crawlers request your pages, which ones are IP-verified, and the URLs they expected to find, including 404s.