Introduction To Incrementality Testing

This is part one of a three-part series on incrementality testing, with subsequent parts delving into more detail on the specifics of conversion lift testing and geo/matched-market testing.

Incrementality testing is growing in importance for media measurement as the protection of Personal Identifiable Information (PII) increases. It already plays, and will continue to play, a critical role in the measurement framework of all agencies and clients looking to understand and quantify the impact of their media campaigns.

As we move into a cookie-less and privacy-first world, attributing performance metrics to ads is becoming increasingly complex. It’s vital, therefore, to understand what incrementality is, how it differs from attribution, and the insights it can provide.

Attribution vs Incrementality Testing

Attribution assesses the contribution of each marketing touchpoint in driving a specific conversion, whereas incrementality testing measures the causal impact of a marketing campaign on conversions that wouldn’t have occurred without it.

There are several attribution methodologies, including last-touch attribution and multi-touch attribution. Multi-touch attribution is now largely data-driven, using probabilistic and machine learning techniques, as we transition away from deterministic measurement. Attribution has historically enabled marketers to understand which touchpoints are most effective at driving conversions. However, as digital tracking capabilities become more limited and methods relying on unique identifiers and third-party cookies become obsolete, the ability to rely solely on attribution for campaign measurement and optimization is waning.

That’s where incrementality testing comes in. It allows digital marketers to adjust to these changes with a future-proofed measurement solution. Using statistical techniques, incrementality testing isolates the true impact of your marketing activities using aggregated data, independent of attribution.

While attribution will remain a core part of your measurement framework, its significance will lessen. It will now be employed exclusively for within-channel, tactical optimizations, such as identifying the most efficient targeting strategy or the highest-engaging creative etc.

The two most relevant incrementality tests for performance marketing are conversion lift tests and geo-tests. Geo-testing doesn’t rely on PII data, making it the ‘gold standard’ in cookie-less measurement. Conversely, in-platform conversion lift tests require significant client data and typically large budgets to provide robust results. Note: AB tests, while valuable, are not true incrementality tests as they do not measure the incremental value of the ad to the business; they only measure the difference in performance between two ads.

Considerations

Pretest setup is a crucial aspect of all statistical tests as it ensures statistically significant and actionable results. Typical pre-test checks include selecting the right audience, the channel, KPI, budget, and minimum detectable lift. Remember, a well-executed setup guarantees a functional test, not necessarily positive results. Given their binary outcome (the campaign had an impact or it didn’t), incrementality tests can risk budgets and don’t always favor media agencies. Therefore, they should not always be the first measurement option suggested.

Finally, being specific about what is being measured, and the level of granularity available for that KPI, is key to success. This could be user engagement, conversion rate, time spent on the website, or any other relevant metric; it’s a balancing act of what can be measured vs. what you want to measure. Generally, the lower down the funnel, the harder it is to measure, the more time required, the more media investment required, and the higher the probability of an unclear outcome. Of course, this must match what the client seeks to measure.