Decoding Incrementality: The Power of Geo-Experimentation

With the ongoing depreciation of user-level tracking, it has and will become harder to understand the impact of our marketing activity. Geo-Experimentation offers a powerful approach to understanding the incrementality of these activities. By testing marketing interventions in designated geographic regions and comparing them to control regions, we can unlock a clearer picture of our marketing's impact. This is critical for those looking at optimising spend and ensuring every dollar is accounted for.

This post is tailored for data-driven marketers, analysts, and decision-makers who are ready to dive deep into the mechanics of Geo-Experimentation to validate the effectiveness of marketing campaigns in a quantifiable manner.

‍

UNDERSTANDING GEO EXPERIMENTATION

‍

Geo-Experimentation has been around for a while, but historically it has been used to understand the impact of out-of-home marketing activity such as TV and radio. It has risen to the forefront of Marketing Analytics over the last few years as companies scramble to find the best way to measure their digital marketing in a cookie-less world. 

The purpose? To assess the real-world impact of marketing strategies or campaigns. 

‍

Step 1: Selecting Treatment and Control Regions with Time-Series Analysis

‍

In identifying "treatment" regions for our marketing interventions, we use a sophisticated time-series analysis. This process involves pairing regions based on their relative trends in pre-defined metrics such as new users, subscribers or sales. The aim is to match treatment regions with control regions (states, zip codes, or any type of locations) that have demonstrated similar behavior over time before the marketing activity starts. The Matched Markets Python library facilitates this by employing a greedy search algorithm, ensuring we apply a methodical selection process that enhances the validity of our experiment's outcomes.

Step 2: Conducting a Power Analysis

‍

Once we know our treatment and control regions we then conduct a power analysis, which is a crucial step before starting the experiment. This step ensures that the scale of the experiment and the budget allocated is sufficient enough to detect a meaningful uplift in the metric of interest. A power analysis sets the stage by linking impression-to-user conversion rates with cost per thousand impressions, allowing us to understand the investment required. 

‍

Step 3: Running the Geo-Experiment and Considering the Cooldown Period

‍

Running the Geo-Experiment can take multiple forms, this could be launching a new channel, campaign or making another significant change in your marketing. When deciding on what you want to test you need to take into consideration how the marketing channel will be impacted by the change, i.e pausing an adset could shift spending to another adset, invalidating the results.

You can then work with channel managers to launch the new campaign or channel, allocating your pre-determined spend over the period agreed upon from the power analysis. The channel or campaign will only be live in the treatment regions and no additional changes will be made to the control regions.

Including a cooldown period is equally important, this duration, following an intervention, allows for us to capture all conversions within the designated attribution window and helps the control and treatment regions return to their natural state after the experiment, setting you up to start your following tests. 

‍

Step 4: Analyzing Results and The Logic Behind the Analysis

‍

When analyzing the results, the logic behind the analysis is as important as the data itself. Matched Markets leverages the time-series trend of the control group to estimate what the results would have been in the treatment group without the intervention (counterfactual), we then combine this with the actual results from the treatment group. The difference—or delta—is then presented as the estimated uplift. 

Once you have reached the end of your cooldown period you will now be able to see the cumulative effect of your marketing activity. As we can measure the uplift in users, subscribers or orders brought in by the marketing activity, and the fact we have the marketing cost from the campaign or channel launch, we can calculate the incremental Cost Pers and ROAS. 

This allows us to understand the incremental impact the channel or campaign is driving, rather than either what the in-platform stats or in-house reporting say.

Geo-Experimentation in Practice

At Cleo, we have developed an in-house tool that integrates the Matched Markets Python library with Streamlit, enabling our teams to autonomously set up, monitor, and review Geo-Experiments. This tool has encouraged ownership among channel managers, creating an environment encouraging experimentation and learning.

Integrating Geo-Experimentation for Holistic Insights

While Geo-Experimentation is an invaluable tool for deciphering the true value of marketing initiatives, it should not be the sole source of insight. When integrated with a data-driven attribution model and a Marketing Mix Model, Geo-Experimentation becomes part of a comprehensive approach. This empowers you to allocate marketing resources more effectively, ultimately driving better business outcomes.

By embracing Geo-Experimentation with a strategic, structured approach, marketers can gain a competitive edge, ensuring that their decisions are grounded in solid, incremental data.

‍