Bots & AI crawlers

Most crawlers don't run the browser script. Send their requests from your server or edge middleware instead.

Send crawler requests

1. Capture the original request

   In your server or edge middleware, read the request's user-agent, URL and response status.

2. POST it with your secret key

   terminal

   Copy

   1

   2

   3

   4

   5

   6

   7

   8

   curl --request POST 'https://metricraven.com/api/v1/crawls' \
     --header "Authorization: Bearer $METRICRAVEN_SECRET_KEY" \
     --header 'Content-Type: application/json' \
     --data '{
     "url": "https://example.com/pricing",
     "userAgent": "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)",
     "status": 200
   }'

   The example simulates a crawler. Replace the user-agent, URL and status with the real values.

MetricRaven classifies supported bots and AI crawlers from the user-agent.