meta-webindexer

User agent

meta-webindexer

MetricRaven matches this token anywhere in the request's user agent, ignoring case. Operators add version numbers and browser text around it, for example "meta-webindexer/1.0".

IP verification

Verified when the user agent and the IP range both match

1. 1User agent

   The request's user agent contains meta-webindexer.

2. 2Request IP

   Your server-side tracker sends the IP that made the request.

3. 3Range check

   The IP is compared with the ranges Meta publishes, refreshed twice a day.

Range sources

• Meta network (AS32934)

  https://stat.ripe.net/data/announced-prefixes/data.json?resource=AS32934

  Meta doesn't publish crawler-specific ranges; this is the network it announces, so a match means the request came from Meta's infrastructure.

Control it in robots.txt

User-agent: meta-webindexer
Disallow: /private/

Well-behaved crawlers re-read robots.txt within a day or so.

Questions

What is meta-webindexer?

Crawls to improve Meta AI search results, so Meta AI can cite and link to pages. MetricRaven groups it under Search indexes: search engines and AI search keeping their index fresh.

How do I verify that a request really comes from meta-webindexer?

Check both the user agent and the IP. Requests from meta-webindexer contain "meta-webindexer" in the user agent and come from the IP ranges Meta publishes (https://stat.ripe.net/data/announced-prefixes/data.json?resource=AS32934). A request with the name but another IP is likely spoofed.

How do I block or allow meta-webindexer?

Add a robots.txt group for "meta-webindexer" with the paths to allow or disallow. Fetchers that act for a user may ignore robots.txt; block those at your server or firewall instead.

How can I see when meta-webindexer visits my website?

Most crawlers never run JavaScript, so browser analytics miss them. MetricRaven's @metricraven/ai-crawl package reports crawler requests from your server, verifies their IPs against published ranges, and shows which pages each crawler reads.