Welcome to the final part of our three-part series on incrementality testing. In this piece, we take a closer look at geo-lift testing and its growing importance within the media industry.
As we have explored in parts one and two of this series, privacy regulations have fundamentally altered how we can measure the impact of media. The fact remains, though, that the ability to measure the impact and ROI of ad campaigns is critical for advertisers. Appropriate measurement facilitates both tactical and strategic decision-making and allows advertisers to prove the commercial benefit of media investment.
So, with deterministic user-level data on the wane our reliance on incrementality testing, which can leverage aggregated performance data, is increasing exponentially. Part two of this series took a closer look at one variety of incrementality testing, the conversion lift test. Whilst perhaps the most statistically robust methodology for determining the causal impact of media on business outcomes, it is rarely recommended as a testing strategy due to both technical and practical feasibility.
Privacy regulations pose significant challenges. For example, the depreciation of IDFA, Apple’s ATT and SKAdNetwork, make it difficult for conversion lift tests to measure the conversion volume from the control group – the group that did not interact with the ad (TikTok currently only offers conversion lift studies for Android users). Additionally, they typically require large amounts of client data and, in some cases, prohibitively large media budgets to generate robust results.
This leads us to geo-tests, otherwise known as matched-market or geo-lift tests. In a privacy-centric world – where there is little to no user-level data – geo-tests are the gold standard, as they rely solely on aggregated KPI data. In brief, a geo-test involves exposing a geographical test region(s) to a chosen media campaign/channel (green), whilst leaving this media absent from a geographical control region(s) (red), then measuring the difference in a chosen business metric. This difference is the incremental value delivered by the media in the test region.
“Geo test is a future proof solution that helps marketers understand the value of their media investment and unpick the incremental impact their media activity.”
Gauthier Rochas, Associate Director, Global,
Data Analytics & Tech, M&C Saatchi Performance
Although geo-tests are not strictly Randomized Control Trials (RCTs1), as with conversion lift tests, they overcome their main drawback: they don’t require conversions to be attributed at the user level, making them resilient to the reduction of deterministic attribution. Additionally, contrary to conversion lift tests, geo-tests do not need to be performed in-platform, giving analysts greater control over the test structure. Ultimately, geo-tests are the most future-proofed incrementality testing method. They offer decision-makers a clear and intuitive understanding of the value of their media investment.
Geo-testing can be broken down into three phases: Pre-test setup; Activation/ testing window; Post-test analysis. Let’s take a closer look at each phase.
- RCT (Randomised Controlled Trial): A test in which each subject taking part in the experiment is randomly assigned to either the test group (exposed to the ad) or the control group (not exposed to the ad). The two groups are tracked and their conversion are recorded. We then analyze the results using statistical methods to understand if the ad exposure drove more conversion. ↩︎
Pre-Test Setup
To refresh, a solid pretest setup is a prerequisite for any successful and actionable incrementality test; this is where most tests fail. Before initiating a geo-test, ensure the following conditions:
#1 Decide On The Media Channel Or Campaign You Want To Test
Aim to isolate the incremental impact of a specific channel/campaign by testing one channel or campaign at a time. Additionally, you must be able to purchase media at a regional level of granularity.
#2 Define Your Campaign Objectives And KPIs
The granularity of your KPI is paramount – it will determine the level of geographical granularity for your test. For example, if your media can be purchased at city-level, but your KPI can only be measured at state-level, you will have to construct your test at state-level.
#3 Gather Data For Your KPI
Typically, 3-6 months of historical data, split at your planned regional granularity and at daily frequency, is enough to establish a baseline of performance for test and control regions.
#4 Test And Control Region Selection
You can choose the region(s) for your test, but your data science team will use various algorithms to select your control region(s). This involves using synthetic control methods, which work by selecting a control unit that is virtually identical to the test unit. By using historical data, we test to find groups with a very high correlation (typically greater than 98%) – if the correlation between test and control regions is 98%, the error margin of the result is 2%. It is vital that the test and control cells behave in the same way across time for the test to be valid – this allows us to attribute any difference in performance between regions to the media. Note, the test and control regions do not need to be the same size as performance metrics will be indexed.
- Your test region(s) will be exposed to your media campaign whereas your control will not.
- Mention all other media that will be active during the test.
- Identify any events that might occur during the test that could impact the target KPI.
Testing Window
Once you’ve completed all pre-test planning steps and launched your test activity, you should be aware of the following:
- The only difference between the test and control regions should be the media being tested: all known variables and media campaigns should be equal in the test and control regions, except for the media channel being tested, so that the incremental impact of the test media can be isolated. If discrepancies arise during the test, it will be invalid.
- The test activity should run for 4-6 weeks – the bare minimum is 2 weeks, but statistical robustness is more likely with a 4–6 week testing window. The shorter the test, the higher the risk of a negative test outcome.
- The campaign should be regularly monitored to ensure the correct data is gathered and that the media campaign is active only in the test region(s).
- After the activity concludes, allow for a 1-2 week listening or cool-off period to capture the tail of your media campaign. This is required to validate that the test and control regions are ‘identical’ and valid for comparative testing. If they do not return to their pre-test relationship, the test is invalidated. The listening period will also allow us to calculate the adstock of your media input.
Post-Test Analysis
Once the test is complete, and all data for your campaign KPI is accurately collected, a statistical analysis is performed to uncover the incremental impact of your media campaign. Similar to conversion lift testing, statistical methods are employed to model the impact of the chosen media on your chosen KPI. With careful testing protocols, the difference between your test and control regions is identified as the incremental impact of your media activity – i.e., media caused this outcome.
Key Takeaways & Conclusions:
Geo-testing offers a reliable and privacy-compliant method for measuring the impact of your media campaigns, particularly in a world where user-level data is increasingly regulated. It provides a nuanced understanding of how your media investments affect chosen business metrics, delivering actionable insights that can guide your decision-making process and, ultimately, improve the ROI of your marketing efforts.
Setting up a geo-test requires careful planning, data collection, and analysis. However, the rewards they offer in actionable insights and informed decision-making make them a worthwhile investment for any advertiser navigating the privacy-centric landscape of the digital age. Understanding how to properly execute geo-tests can enhance the effectiveness of your campaigns, drive better results, and raise your marketing strategy to new standards.
- Privacy-Compliant: Geo-tests respect user privacy by using aggregated KPI data, aligning with the recent wave of privacy regulations and limitations on user-level data.
- Future-Proof: Unlike conversion lift tests, geo-tests do not rely on user-level attribution. If your media and KPIs can be purchased or measured at a certain regional level, geo-tests remain a viable and future-ready option.
- Pre-Test Planning: It’s crucial to set the right foundations with a thorough and well-executed pre-test setup. This stage defines the accuracy and validity of your subsequent results. By ensuring you’ve meticulously planned and prepared, you can maximize the potential of your geo-test.
- Proves Incrementality/ Causation: Geo-tests offer a way to understand the true impact of your marketing investments. They prove that media/ marketing was responsible for the outcome. It’s important to note that certain channels, like Paid Search, can sometimes underperform in incrementality tests. However, whilst some channels might not show substantial incremental benefits, they are still integral to the overall marketing mix. Specifically, Paid Search operates more like a storefront than a traditional media channel, thus might not always align with typical performance expectations.
- Validates Other Approaches: Geo-tests often act as “ground truth” for other measurement approaches such as Marketing Mix Modeling (MMM). This ensures you gain accurate insights not only from geo-tests but also from your broader marketing analytics ecosystem.
- Plan and Test Often: Geo-testing isn’t a one-and-done affair. Implement a strategy that incorporates multiple tests across various channels throughout the year. Regular testing allows for continued optimization and refinement of your marketing strategy.
