meta-externalfetcher
Meta · AI answers · User agent + IP range
About this crawler
Fetches individual links at a user's request, including for Meta AI agents completing tasks. May bypass robots.txt.
Usually means: Someone asked an AI assistant a question and it read this page to answer, often with a link back.
User agent
meta-externalfetcher
MetricRaven matches this token anywhere in the request's user agent, ignoring case. Operators add version numbers and browser text around it, for example "meta-externalfetcher/1.0".
IP verification
Verified when the user agent and the IP range both match
1User agent
The request's user agent contains
meta-externalfetcher.2Request IP
Your server-side tracker sends the IP that made the request.
3Range check
The IP is compared with the ranges Meta publishes, refreshed twice a day.
Range sources
Meta network (AS32934)
https://stat.ripe.net/data/announced-prefixes/data.json?resource=AS32934
Meta doesn't publish crawler-specific ranges; this is the network it announces, so a match means the request came from Meta's infrastructure.
Control it in robots.txt
User-agent: meta-externalfetcher Disallow: /private/
Fetchers that act for a person may not follow robots.txt; block them at your server or firewall if you need to.
Questions
- What is meta-externalfetcher?
- Fetches individual links at a user's request, including for Meta AI agents completing tasks. May bypass robots.txt. MetricRaven groups it under AI answers: assistants opening a page because a person asked about it.
- How do I verify that a request really comes from meta-externalfetcher?
- Check both the user agent and the IP. Requests from meta-externalfetcher contain "meta-externalfetcher" in the user agent and come from the IP ranges Meta publishes (https://stat.ripe.net/data/announced-prefixes/data.json?resource=AS32934). A request with the name but another IP is likely spoofed.
- How do I block or allow meta-externalfetcher?
- Add a robots.txt group for "meta-externalfetcher" with the paths to allow or disallow. Fetchers that act for a user may ignore robots.txt; block those at your server or firewall instead.
- How can I see when meta-externalfetcher visits my website?
- Most crawlers never run JavaScript, so browser analytics miss them. MetricRaven's @metricraven/ai-crawl package reports crawler requests from your server, verifies their IPs against published ranges, and shows which pages each crawler reads.
See when AI assistants and crawlers read your site
Add one line to your server and MetricRaven shows which assistants, search engines and training crawlers request your pages, which ones are IP-verified, and the URLs they expected to find, including 404s.