What incrementality means
Every attribution model answers "which touch gets credit for this sale?" Incrementality asks a harder question: "would this sale have happened without the ad?" A retargeting ad shown to someone who already had the product in their cart may get full credit in an ad platform and still have caused nothing.
Incrementality is measured with an experiment. You split people, or regions, into a test group that gets the marketing and a control group that doesn't. The difference in results between the two groups is the incremental effect. Everything else, the sales the control group made on its own, is the baseline.
It matters most where attribution is most generous: retargeting, branded search, view-through conversions and campaigns aimed at existing customers. These tend to look excellent in attribution reports and can show small incremental lift, because the people they reach were already likely to buy.
Incrementality formulas
Lift = (test conversion rate − control conversion rate) ÷ control conversion rate
Incremental conversions = test conversions − (control conversion rate × test group size)
Incremental ROAS = incremental revenue ÷ ad spend
Lift tells you how much better the test group did in relative terms. Incremental conversions turn that into a count you can put a cost on. Incremental ROAS, or cost per incremental conversion, is the number to compare against the ROAS your attribution reports.
Worked example
You run a retargeting campaign for a month and hold out a random 10% of the audience, who never see the ads.
| Group | People | Buyers | Conversion rate |
|---|---|---|---|
| Test (saw ads) | 90,000 | 2,250 | 2.50% |
| Control (holdout) | 10,000 | 200 | 2.00% |
Lift = (2.50% − 2.00%) ÷ 2.00% = 25%
Without the ads, the test group would have produced about 90,000 × 2.00% = 1,800 buyers. So 2,250 − 1,800 = 450 buyers were incremental. The ad platform credits the campaign with far more of the 2,250.
At an average order of $80 and $15,000 of spend, attributed ROAS on all 2,250 buyers would be 2,250 × $80 ÷ $15,000 = 12. Incremental ROAS is 450 × $80 ÷ $15,000 = 2.4. With a break-even ROAS of 2, the campaign still pays, but far less than the report suggests.
Ways to test incrementality
- User-level holdout. The ad platform randomly withholds ads from part of the audience and compares the groups. It is the cleanest design, but it relies on the platform's own conversion data.
- Geo tests. You turn a channel on or off, or change spend, in some regions and keep others as controls, then compare revenue by region from your own data. They work for any channel, including TV and podcasts, and need no user tracking. They need enough regions and volume to separate the effect from noise.
- On/off tests over time. Pause a channel for a few weeks and watch total revenue. Cheap, but seasonality, launches and other campaigns muddy the result. Use it as a rough check, not proof.
- Model-based estimates. Media mix modeling estimates incremental effects from spend and sales over time. Some ad platforms now model it too: Meta offers an incremental attribution model that optimizes and reports on conversions its models predict the ad caused.
How to run a good test
- Decide the metric before you start: new customers, revenue or MER, measured from your payment data.
- Make the groups comparable. Randomize people, or match regions on past revenue.
- Run long enough to cover your buying cycle. If customers take two weeks to decide, a one-week test misses most of the effect.
- Change one thing at a time, and write down when it started and ended.
- Read the result as a range, not a single number. Small tests produce wide ranges.
Common incrementality mistakes
- Treating attributed conversions as incremental. Attribution divides credit; it doesn't measure cause.
- Tests that are too small. A lift of 5% on a few hundred conversions is indistinguishable from chance.
- Contaminated controls. If the holdout group sees the same offer in email or on other networks, the lift shrinks.
- Testing once and assuming it holds forever. Lift changes with audience, creative and spend level. Retest when any of them changes a lot.
How MetricRaven fits in
MetricRaven doesn't run incrementality tests or calculate lift. It gives you the measurement side from your own payments, which is what a geo test or an on/off test needs:
- Revenue and purchase rate by country and region, so you can compare test and control regions on real payments, not pixel conversions.
- Compare against the previous period, matching weekdays or the same period last year, and chart notes to mark when a test started and stopped.
- First-touch revenue, customers and ROAS per Meta and Google Ads campaign, with the purchase value Meta reports for itself shown alongside for comparison.
See Chart notes, Filters & segments and the guide to measuring paid ads revenue.
Questions
- What is incrementality in marketing?
- It is the part of your results that a campaign actually caused. You measure it by comparing people or regions that saw the campaign with a similar group that didn't. The difference is the incremental effect.
- How do you calculate incrementality?
- Compare conversion rates of a test group and a control group. Lift = (test rate − control rate) ÷ control rate. Incremental conversions = test conversions − control rate × test group size.
- What is the difference between attribution and incrementality?
- Attribution assigns credit for sales that happened to the touches before them. Incrementality measures how many of those sales the marketing caused. A campaign can get a lot of attributed credit and still have low incremental lift.
- What is a geo test?
- An incrementality test where you change marketing in some regions and keep others as a control, then compare sales by region. It doesn't need user-level tracking, so it works for any channel.