Track AI crawlers and traffic from AI assistants

People now ask ChatGPT, Claude and Perplexity for recommendations instead of searching. Before an assistant can mention you, its crawlers have to read your pages. Both halves are worth tracking: what AI reads, and who clicks through.

Why this matters

If AI assistants never fetch your pricing or docs pages, they can't recommend you accurately. And if visitors from chatgpt.com convert better than search visitors, that's a channel worth writing for.

Why most analytics tools can't tell you

Crawlers don't run JavaScript, so browser-based analytics never sees them; they only show up in server logs. Visits from AI assistants are usually lumped in with all other referrals, with no revenue next to them.

How MetricRaven does it

MetricRaven accepts crawler requests from your server or edge middleware and classifies them by user-agent: AI answers (assistants and AI search such as ChatGPT-User, OAI-SearchBot and PerplexityBot), search indexing (Googlebot, Bingbot), AI training (GPTBot, ClaudeBot) and other bots. It shows the pages each crawler fetched and requests for robots.txt and llms.txt. Bot requests don't count toward your monthly events. People who click through from an assistant appear in your sources like any other referrer, with their revenue.

How to do it

1. In your server or edge middleware, capture each request's user-agent, URL and response status.
2. POST it to the crawls endpoint with your site's secret key, from your backend only (Bots & AI crawlers).
3. Open Traffic and scroll to Bots & AI crawlers to see requests by crawler, purpose and page.
4. In Acquisition → Referrer, look for chatgpt.com, perplexity.ai and similar hosts, sorted by revenue.
5. Click a referrer to filter the dashboard and see which pages those visitors land on.

Ask your own analytics questions from Claude or ChatGPT too: the MCP server can report crawler activity alongside traffic.