Googlebot

User agent

Googlebot

MetricRaven matches this token anywhere in the request's user agent, ignoring case. Operators add version numbers and browser text around it, for example "Googlebot/1.0".

IP verification

Verified when the user agent and the IP range both match

1. 1User agent

   The request's user agent contains Googlebot.

2. 2Request IP

   Your server-side tracker sends the IP that made the request.

3. 3Range check

   The IP is compared with the ranges Google publishes, refreshed twice a day.

Range sources

• Google common crawlers

  https://developers.google.com/static/crawling/ipranges/common-crawlers.json

Control it in robots.txt

User-agent: Googlebot
Disallow: /private/

Well-behaved crawlers re-read robots.txt within a day or so.

Questions

What is Googlebot?

Google Search's crawler. Its crawls also feed AI features in Search such as AI Overviews. MetricRaven groups it under Search indexes: search engines and AI search keeping their index fresh.

How do I verify that a request really comes from Googlebot?

Check both the user agent and the IP. Requests from Googlebot contain "Googlebot" in the user agent and come from the IP ranges Google publishes (https://developers.google.com/static/crawling/ipranges/common-crawlers.json). A request with the name but another IP is likely spoofed.

How do I block or allow Googlebot?

Add a robots.txt group for "Googlebot" with the paths to allow or disallow. Fetchers that act for a user may ignore robots.txt; block those at your server or firewall instead.

How can I see when Googlebot visits my website?

Most crawlers never run JavaScript, so browser analytics miss them. MetricRaven's @metricraven/ai-crawl package reports crawler requests from your server, verifies their IPs against published ranges, and shows which pages each crawler reads.