Bot traffic settings

Decide which crawlers MetricRaven records, keep your secret key out of the middleware that reports them, and see how many bot requests your plan has left.

Exclude companies or agents

In Settings → Tracking & install → Bot traffic, turn off a whole company (for example Meta) or a single agent (for example GPTBot). Excluded crawl events are dropped when they arrive: they aren't stored, shown in reports or counted toward your allowance. The change applies to new requests only; past data stays as it was.

Admins and the owner can change these settings. Viewers can see them. Names match without regard to case, and you can exclude agents MetricRaven doesn't list yet by name.

Bot token

The bot token (mrbot_…) can only send crawl events to POST /api/v1/crawls. It can't read reports, send payments or change settings, so it's the safer credential for edge or server middleware.

1. Create the token

   Create it under Bot traffic. It's shown once; only a hash is stored.

2. Send crawl events with it

   With @metricraven/ai-crawl or the Node.js SDK, set METRICRAVEN_BOT_TOKEN and you're done. With the API:

   terminal

   Copy

   1

   2

   3

   4

   5

   6

   7

   8

   9

   curl --request POST 'https://metricraven.com/api/v1/crawls' \
     --header "Authorization: Bearer $METRICRAVEN_BOT_TOKEN" \
     --header 'Content-Type: application/json' \
     --data '{
     "url": "https://example.com/pricing",
     "userAgent": "Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)",
     "status": 200,
     "ip": "203.0.113.7"
   }'

   ip is optional. You can add siteId (or ?site=) to check the token belongs to that website.

3. Optionally require it

   Turn on Only accept crawl events sent with the bot token to refuse crawl events sent with the secret key (HTTP 403, bot_token_required). Deploy the token first.

Rotating the token replaces it at once, and the old token stops working. Deleting it also turns off the requirement, so crawl events sent with the secret key are accepted again.

Record verified crawlers only

When a crawl event carries the request's IP, MetricRaven checks it against the IP ranges the crawler's operator publishes (see IP verification). Turn on Record verified crawlers only to drop events whose IP is outside those ranges: someone sending a crawler's user agent from another network. They return ignored: "unverified" and aren't stored or counted.

Crawl event responses

Bot request allowance

Each plan includes a monthly allowance of AI and bot requests, pooled across your websites and separate from your events. You get an email at 80% and when it runs out. At the limit, new crawl events aren't recorded until the cycle resets. Nothing extra is charged, and pageviews, goals and payments keep working. Usage is shown under Billing & usage, and excluding crawlers you don't need stretches the allowance.

API, CLI and MCP

These endpoints act as a person: use an account access token with ?site=, not the secret key or the bot token. Changes need the write scope from an admin or the owner.

curl --request PATCH 'https://metricraven.com/api/v1/bot-traffic/settings?site=YOUR_SITE_ID' \
  --header "Authorization: Bearer $METRICRAVEN_TOKEN" \
  --header 'Content-Type: application/json' \
  --data '{ "disableCompanies": ["Meta"], "disableAgents": ["GPTBot"] }'

CLI: metricraven bots settings, bots usage, bots exclude GPTBot, bots include Meta --company, bots require-token on. MCP: get_bot_settings, get_bot_usage and update_bot_settings (which asks you to confirm first).