Incrementality Testing
Incrementality testing measures whether a marketing campaign actually causes additional conversions or simply captures conversions that would have happened anyway. For performance marketing, this is the most rigorous way to evaluate channel effectiveness because it separates correlation from causation, going beyond what attribution models can tell you.
This page covers how incrementality testing works, the methods available, common mistakes, and my practical experience designing and running these experiments.
What It Is and Why It Matters
The fundamental question incrementality testing answers is: "What would have happened if we had not run this campaign?" If a user clicks a brand search ad and then purchases, did the ad cause the purchase, or would the user have purchased anyway by clicking the organic result below it?
Attribution models cannot answer this question because they observe correlations between touchpoints and conversions. Incrementality testing uses experimental methods (test vs control groups) to measure causation.
The two main approaches are: holdout experiments (showing ads to a test group while withholding them from a control group, then comparing conversion rates) and geo lift tests (running campaigns in some geographic regions while holding them out of comparable regions, then measuring the difference in business outcomes).
For performance marketing teams, incrementality testing is the gold standard for evaluating channel value. It is especially important for brand search (which often captures rather than creates demand), retargeting (which targets users already close to converting), and broad awareness channels (where attribution models struggle to assign credit accurately).
The challenge is that incrementality testing requires statistical rigor, sufficient sample sizes, and the willingness to sacrifice some short term revenue by withholding campaigns from control groups. But the insights it provides can save organizations significant wasted spend on channels that appear effective in attribution but do not actually drive incremental conversions.
Common Use Cases
How incrementality testing is applied in marketing.
Brand Search Lift Measurement
Testing whether brand search ads drive incremental conversions or primarily capture users who would have clicked organic results. This is one of the most common and highest impact incrementality tests because brand search often represents significant spend.
Retargeting Effectiveness
Measuring whether retargeting ads cause additional purchases or target users who were already going to convert. Retargeting typically shows high ROAS in attribution but may have lower incrementality because it reaches users already in the conversion funnel.
Geo Lift Testing
Running campaigns in test regions while holding comparable control regions without ads. Comparing business outcomes (revenue, conversions, store visits) between regions to measure the incremental impact of advertising at the market level.
Budget Optimization
Using incrementality data to reallocate budget from low incrementality channels to high incrementality channels. This often means shifting spend from brand search and retargeting toward prospecting and awareness activities that attribution models undervalue.
New Channel Evaluation
Testing whether a new marketing channel (Connected TV, podcast ads, influencer partnerships) drives actual incremental business impact beyond what existing channels already provide.
Audience Segment Testing
Measuring whether advertising to specific audience segments drives incremental conversions from that segment. Determining which targeting strategies add value versus which simply identify users who would convert regardless.
Practical Experience
I have designed and run incrementality tests across multiple channels and industries. The tests that consistently produce the most actionable insights are brand search holdouts and retargeting holdouts, because these channels tend to have the largest gap between attributed and incremental performance.
For brand search, the typical finding is that 60 to 80 percent of conversions attributed to brand search ads would have occurred organically. This does not mean brand search ads are worthless. The remaining 20 to 40 percent represents genuine incremental value, and the ads also protect against competitor conquesting. But it does mean the true cost per incremental conversion from brand search is 3 to 5 times higher than the attributed CPA suggests.
For retargeting, incrementality typically ranges from 20 to 50 percent depending on the remarketing strategy. Broad retargeting (everyone who visited the site) tends to have lower incrementality than narrow retargeting (users who abandoned cart with items). This insight helps optimize remarketing strategy: focus retargeting spend on user segments where the incremental impact is highest.
Geo lift tests are my preferred method for measuring upper funnel channels (display, video, social awareness) where user level holdout experiments are difficult or impossible. I select matched geographic pairs based on historical performance, population, and demographic similarity. The test runs for 4 to 8 weeks, and I use statistical methods to account for baseline differences and external factors.
The practical challenge with incrementality testing is organizational. It requires deliberately withholding marketing spend from some users or regions, which creates short term revenue risk. I address this by starting with small scale tests (5 to 10 percent holdout groups) and building evidence gradually. Once stakeholders see the data, they typically support broader testing because the budget reallocation insights are so valuable.
I combine incrementality results with attribution analysis in BigQuery to calibrate attribution models. If a channel shows 500 attributed conversions but incrementality testing suggests only 200 are incremental, I apply that calibration factor to the attribution model for more accurate ongoing reporting.
Common Mistakes to Avoid
Common incrementality testing mistakes that produce unreliable results.
Test Groups Too Small
Running tests with insufficient sample sizes, leading to statistically insignificant results. Before starting a test, calculate the required sample size based on expected effect size and desired confidence level. Underpowered tests waste time and produce unreliable conclusions.
Contaminated Control Groups
Allowing test channel exposure to leak into control groups. For holdout experiments, ensure the ad platform actually withholds ads from the control group. For geo tests, ensure campaigns in test regions do not spill into control regions through audience targeting that crosses geographic boundaries.
Testing for Too Short a Period
Running tests for only 1 to 2 weeks. Short test periods are affected by weekly seasonality, promotional events, and random variation. Most incrementality tests need 4 to 8 weeks to produce reliable results. Plan test duration based on your conversion volume and typical purchase cycle.
Testing Everything at Once
Running multiple overlapping tests simultaneously, making it impossible to isolate the impact of each channel. Test one channel or one variable at a time. Sequential testing with clear gaps between tests produces cleaner results.
Not Acting on Results
Running incrementality tests and then not changing budget allocation based on the findings. The entire purpose of testing is to inform decisions. If a test shows that a channel has low incrementality, reallocate that budget to channels with higher incremental impact.
Frequently Asked Questions
A/B testing compares two versions of something (ad creative, landing page, email subject line) to see which performs better. Incrementality testing compares advertising versus no advertising to measure whether the advertising causes additional conversions. A/B testing optimizes within a channel. Incrementality testing evaluates whether the channel itself adds value.
You need enough conversion volume to achieve statistical significance. As a rough guide, you need at least 100 conversions in both test and control groups during the test period. For channels with lower conversion rates, this means larger audiences and longer test periods. Start with your highest spend channels where even a small incrementality finding has significant budget implications.
Yes. Meta offers conversion lift studies through its platform, which are essentially holdout experiments. You can also run geo lift tests by restricting Meta campaigns to specific regions and comparing against control regions. For the most rigorous results, work with Meta partner measurement solutions or build custom testing frameworks that control for external factors.
It depends heavily on the channel and campaign type. Prospecting campaigns on new audiences typically show 50 to 80 percent incrementality. Retargeting campaigns typically show 20 to 50 percent. Brand search ads typically show 20 to 40 percent. These ranges vary significantly by industry, brand strength, and competitive environment. The absolute number matters less than using the data to compare channels and allocate budget.
Test each major channel at least once per year, and retest whenever you make significant changes to campaign strategy, targeting, or budget levels. Market conditions, competition, and consumer behavior change over time, so historical incrementality results should be refreshed periodically. Quarterly testing cadence for your top 2 to 3 channels is a good rhythm for mature marketing programs.
Related Topics
Performance Marketing
Paid acquisition strategy.
Marketing Attribution
Attribution models and analysis.
Google Ads
Search advertising.
Meta Ads
Social advertising.
BigQuery
Data warehouse for analysis.
Data Driven Marketing
Using data for decisions.
Marketing Analytics Manager
Data driven marketing.
A/B Testing Explained
Experimentation fundamentals.
Need Incrementality Testing?
I design and run incrementality experiments that measure true campaign impact and optimize budget allocation based on causal evidence, not just correlations.