# Bot traffic settings

> Choose which crawler companies and agents are recorded, send crawl events with an ingest-only bot token, and track the bot request allowance.

Source: https://metricraven.com/docs/bot-traffic-settings  
Markdown: https://metricraven.com/docs/bot-traffic-settings.md  
Section: Features

Decide which crawlers MetricRaven records, keep your secret key out of the middleware that reports them, and see how many bot requests your plan has left.

## Exclude companies or agents

In **Settings → Tracking & install → Bot traffic**, turn off a whole company (for example Meta) or a single agent (for example GPTBot). Excluded crawl events are dropped when they arrive: they aren't stored, shown in reports or counted toward your allowance. The change applies to new requests only; past data stays as it was.

Admins and the owner can change these settings. Viewers can see them. Names match without regard to case, and you can exclude agents MetricRaven doesn't list yet by name.

## Bot token

The bot token (`mrbot_…`) can only send crawl events to `POST /api/v1/crawls`. It can't read reports, send payments or change settings, so it's the safer credential for edge or server middleware.

1. **Create the token**

   Create it under Bot traffic. It's shown once; only a hash is stored.

2. **Send crawl events with it**

   With [@metricraven/ai-crawl](https://metricraven.com/docs/bots-ai-crawlers) or the [Node.js SDK](https://metricraven.com/docs/node-sdk#bot-token), set `METRICRAVEN_BOT_TOKEN` and you're done. With the API:

   ```bash
   curl --request POST 'https://metricraven.com/api/v1/crawls' \
     --header "Authorization: Bearer $METRICRAVEN_BOT_TOKEN" \
     --header 'Content-Type: application/json' \
     --data '{
     "url": "https://example.com/pricing",
     "userAgent": "Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)",
     "status": 200,
     "ip": "203.0.113.7"
   }'
   ```

   `ip` is optional. You can add `siteId` (or `?site=`) to check the token belongs to that website.

3. **Optionally require it**

   Turn on **Only accept crawl events sent with the bot token** to refuse crawl events sent with the secret key (HTTP 403, `bot_token_required`). Deploy the token first.

Rotating the token replaces it at once, and the old token stops working. Deleting it also turns off the requirement, so crawl events sent with the secret key are accepted again.

## Record verified crawlers only

When a crawl event carries the request's IP, MetricRaven checks it against the IP ranges the crawler's operator publishes (see [IP verification](https://metricraven.com/docs/bots-ai-crawlers#verification)). Turn on **Record verified crawlers only** to drop events whose IP is outside those ranges: someone sending a crawler's user agent from another network. They return `ignored: "unverified"` and aren't stored or counted.

> **Note:** Crawlers whose operator publishes no ranges, and events sent without an IP, are still recorded as unverified. Before turning this on, check in **All crawlers** that requests from GPTBot or Googlebot show as IP-verified: if your server forwards a proxy's address instead of the crawler's, every check fails.

## Crawl event responses

| Response | Meaning |
| --- | --- |
| 200 { ok: true, bot, type, verified } | Recorded and counted. verified: true or false after an IP check, null when none was possible. |
| 200 { ignored: "not_a_crawler" } | A normal browser user agent. Nothing recorded. |
| 200 { ignored: "excluded", excludedBy } | The company or agent is excluded. Nothing recorded or counted. |
| 200 { ignored: "unverified" } | Verified only is on and the IP is outside the crawler's published ranges. Nothing recorded or counted. |
| 401 unauthorized | Missing or unknown token or secret key. |
| 402 bot_tracking_paused | The allowance is used up for this cycle. Recording resumes at resetAt. |
| 403 bot_token_required | The website only accepts crawl events sent with its bot token. |
| 403 domain_not_allowed / wrong_site | The URL isn't on the website's domains, or the token is for another website. |

## Bot request allowance

Each plan includes a monthly allowance of AI and bot requests, pooled across your websites and separate from your events. You get an email at 80% and when it runs out. At the limit, new crawl events aren't recorded until the cycle resets. Nothing extra is charged, and pageviews, goals and payments keep working. Usage is shown under [Billing & usage](https://metricraven.com/app/billing), and excluding crawlers you don't need stretches the allowance.

## API, CLI and MCP

These endpoints act as a person: use an [account access token](https://metricraven.com/docs/api-authentication) with `?site=`, not the secret key or the bot token. Changes need the write scope from an admin or the owner.

| Request | Does |
| --- | --- |
| GET /api/v1/bot-traffic/settings?site=… | Exclusions, token status and the catalog of known companies and agents. |
| PUT /api/v1/bot-traffic/settings?site=… | Replaces excludedCompanies and excludedAgents (and optionally rejectWithoutToken and verifiedOnly). |
| PATCH /api/v1/bot-traffic/settings?site=… | Incremental: disableCompanies, enableCompanies, disableAgents, enableAgents, rejectWithoutToken, verifiedOnly. Other exclusions are kept. |
| GET /api/v1/bot-traffic/token?site=… | Token status: hint, createdAt, lastUsedAt. Never the token. |
| POST /api/v1/bot-traffic/token?site=… | Creates the token and returns it once. |
| PUT /api/v1/bot-traffic/token?site=… | Rotates it. The old token stops working at once. |
| DELETE /api/v1/bot-traffic/token?site=… | Deletes it and turns off the requirement. |
| GET /api/v1/bot-traffic/usage?site=… | Requests used, allowance, remaining, refused, resetAt and websites that used them. |

```bash
curl --request PATCH 'https://metricraven.com/api/v1/bot-traffic/settings?site=YOUR_SITE_ID' \
  --header "Authorization: Bearer $METRICRAVEN_TOKEN" \
  --header 'Content-Type: application/json' \
  --data '{ "disableCompanies": ["Meta"], "disableAgents": ["GPTBot"] }'
```

CLI: `metricraven bots settings`, `bots usage`, `bots exclude GPTBot`, `bots include Meta --company`, `bots require-token on`. MCP: `get_bot_settings`, `get_bot_usage` and `update_bot_settings` (which asks you to confirm first).

> **Note:** Sending both a full list and incremental changes for the same list, or enabling and disabling the same name at once, returns 400 conflicting_update.

## Details

- At most 200 excluded companies and 200 excluded agents per website.
- Demo websites have no bot traffic settings.