AI crawlers, search bots and training agents
The crawlers MetricRaven recognizes: what each one is for, how it identifies itself, and where its operator documents it. 48 of them can be verified against IP ranges their operator publishes.
Want to see them on your own site? Track bots and AI crawlers.
84 crawlers found
AI answers
User-triggered · 15 crawlersAssistants opening a page because a person asked about it.
OpenAIChatGPT-UserFetches a page when ChatGPT or a custom GPT opens it to answer a person's question.IP ranges
AnthropicClaude-UserFetches a page when Claude opens it to answer a person's question.IP ranges
PerplexityPerplexity-UserFetches pages for Perplexity answers and the sources it cites, at a user's request.IP ranges
MistralMistralAI-UserFetches a page when Mistral's assistant (Vibe) opens it to answer a question and cite it.IP ranges
DuckDuckGoDuckAssistBotCrawls pages in real time for DuckDuckGo's AI-assisted answers, which cite their sources. Not used for training.IP ranges
GoogleGoogle-AgentAgents hosted on Google infrastructure that browse the web and act on a user's request.IP ranges
AmazonAmzn-UserFetches live information for user actions, such as answering an Alexa question that needs up-to-date facts.IP ranges
Metameta-externalfetcherFetches individual links at a user's request, including for Meta AI agents completing tasks. May bypass robots.txt.IP ranges
Moonshot AIKimi-UserHandles user-initiated actions in Kimi, such as summarizing an article or answering with live web results.IP ranges
xAIGrok-DeepSearchReported as the fetcher behind Grok's DeepSearch answers.Observed token
AlibabaQwen-UserReported as the user-triggered fetcher behind Qwen answers.Observed token
Search indexes
Discovery · 21 crawlersSearch engines and AI search keeping their index fresh.
You.comYouBotCrawls and indexes pages for the You.com search engine.IP ranges
MicrosoftBingbotMicrosoft's crawler for Bing Search, whose index also grounds Copilot answers.IP ranges
AppleApplebotApple's crawler for Spotlight, Siri and Safari search; Apple may also use its crawls to train foundation models.IP ranges
YandexYandexBotYandex Search's main crawler.
BaiduBaiduspiderBaidu Search's crawler.
SeznamSeznamBotThe crawler for Seznam.cz, the Czech search engine.
YahooYahoo SlurpYahoo's search crawler.
ByteDanceTikTokSpiderReported as ByteDance's crawler for TikTok search and the external pages linked from TikTok.Observed token
Training crawlers
Model data · 23 crawlersCrawlers collecting public pages that may be used to train AI models.
Common CrawlCCBotCommon Crawl's crawler, which builds an open web dataset that many AI models are trained on.IP ranges
Coherecohere-training-data-crawlerReported as the crawler that downloads training data for Cohere's language models.Observed token
DiffbotDiffbotCrawls pages to extract structured data for Diffbot's Knowledge Graph and APIs.
Allen InstituteAI2BotThe Allen Institute for AI's crawler, which finds documents for its open AI research and models.
DeepSeekDeepSeekBotReported as DeepSeek's crawler for its models.Observed token
Zhipu AIChatGLM-SpiderReported as Zhipu AI's crawler for its GLM models.Observed token
Other bots
Ads, previews, SEO · 25 crawlersAd checks, link previews, feed readers, SEO tools and other bots.
AhrefsAhrefsBotCrawls for Ahrefs' backlink index and SEO tools.IP ranges
SemrushSemrushBotCrawls for Semrush's SEO and backlink tools.
MajesticMJ12botMajestic's crawler for its link index.
MozDotBotMoz's crawler for its link index.
DataForSEODataForSeoBotDataForSEO's crawler for its backlink data.
XTwitterbotFetches shared links to build X (Twitter) cards.
LinkedInLinkedInBotFetches shared links to build LinkedIn post previews.
SlackSlackbotFetches shared links to unfurl them in Slack.
DiscordDiscordbotFetches shared links to build Discord embeds.Observed token
TelegramTelegramBotFetches shared links to build Telegram previews.Observed token
See when AI assistants and crawlers read your site
Add one line to your server and MetricRaven shows which assistants, search engines and training crawlers request your pages, which ones are IP-verified, and the URLs they expected to find, including 404s.