MetricRaven

What is archive.org_bot?

Internet Archive · Other bots · User agent only

The Internet Archive's Archive-It crawler, which collects pages for the web archives its partner organizations build.

A visit usually means: A product is checking the page for a specific job: an ad review, a link preview, a feed or an SEO index.

archive.org_bot user agent

User-Agent token
archive.org_bot

Internet Archive documents the token but not a full string. Requests contain archive.org_bot, usually with a version number such as “archive.org_bot/1.0” and browser text around it. Match the token anywhere in the user agent, ignoring case.

Paste a string from your logs into the user agent parser to see which crawler it is. New to the term? Read what a user agent is.

Block or allow archive.org_bot in robots.txt

Internet Archive says archive.org_bot follows robots.txt. Archive-It follows robots.txt by default, but the organizations running a crawl can choose to ignore it.

Block everywhere
robots.txt
User-agent: archive.org_bot
Disallow: /
Keep out of a folder
robots.txt
User-agent: archive.org_bot
Disallow: /private/
Allow
robots.txt
User-agent: archive.org_bot
Allow: /

A crawler follows the most specific group that names it, so a rule here overrides your User-agent: * group for this crawler. Well-behaved crawlers re-read robots.txt within about a day.

archive.org_bot IP ranges and verification

Internet Archive publishes no IP ranges for archive.org_bot, so requests are identified by user agent only and recorded as unverified. Anyone can send this user agent.

What archive.org_bot means for your traffic and revenue

archive.org_bot checks pages for one product job, such as an ad review or a feed. It isn't a visitor and shouldn't be counted as one.

Most crawlers never run JavaScript, so browser analytics can't see them, and the ones that do shouldn't count as visitors. MetricRaven's tracker ignores archive.org_bot, and server-side crawl tracking shows which pages it reads instead. See Bots & AI crawlers.

Questions

What is archive.org_bot?
The Internet Archive's Archive-It crawler, which collects pages for the web archives its partner organizations build. MetricRaven groups it under Other bots: ad checks, link previews, feed readers, SEO tools and other bots.
What is the archive.org_bot user agent?
Requests contain the token "archive.org_bot", usually with a version number such as "archive.org_bot/1.0". Match the token anywhere in the user agent, ignoring case.
How do I block archive.org_bot in robots.txt?
Add "User-agent: archive.org_bot" followed by "Disallow: /" to your robots.txt. Internet Archive says archive.org_bot follows robots.txt. Archive-It follows robots.txt by default, but the organizations running a crawl can choose to ignore it.
Does archive.org_bot publish IP ranges?
No. Internet Archive publishes no IP ranges for archive.org_bot, so only the user agent identifies it, and anyone can send that string.
Is archive.org_bot safe?
The real archive.org_bot only requests public pages, and Internet Archive documents it. The risk is impostors using its name. Whether to allow it depends on what you get back: a product is checking the page for a specific job: an ad review, a link preview, a feed or an SEO index.

See when AI assistants and crawlers read your site

Add one line to your server and MetricRaven shows which assistants, search engines and training crawlers request your pages, which ones are IP-verified, and the URLs they expected to find, including 404s.