The Incrementality checklist – what to know before getting started

By Rokt

By: Thi Thumasathit, Head of Product – Campaigns in Product and Design (Sydney)

Over the course of this Incrementality series, we have equipped you with knowledge about the Incrementality methodology and introduced the ‘Wouldve’s’ and how and what Incrementality can measure. Now the time has come to dish out some final tips on how to really get yourselves situated with Incrementality.

Ask your digital marketing platform if they have an Incrementality solution.

Because the Incrementality methodology is so deeply embedded within a platform’s campaign/ad selection process, Incrementality solutions are platform-specific. It’s not as simple as going out and renting or buying an Incrementality solution and use it to measure campaign effectiveness across all your digital marketing platforms.


Be patient.

Like all scientific experiments, Incrementality studies require time and data to produce concrete results. Work with your digital marketing platform to estimate how long it will take for your Incrementality study to reach statistical significance.

Define your key questions.

Define what key questions you’re trying to answer with an Incrementality study.

Most brands start off with the highest level question: what incremental impact can this digital marketing platform having on overall revenue?

Some brands ask more specific questions: what incremental impact can this digital marketing platform have on one specific customer segment?

It’s important to identify what these specific questions are and answer them upfront:

– The questions might affect how campaigns are structured. For example, if your really want to know the impact a given platform is having during the Holiday season, you need to make sure you have campaigns running on that platform during the Holiday season.

– The questions might be limited by what data is/isn’t tracked in the platform. For example, if the platform doesn’t track gender, you can’t ask gender-related questions.

– Keep in mind that the more questions you ask, the smaller the data sets become, and the longer it will take to reach statistically significant answers. Focus on the most important questions at hand.

Recognize what Incrementality is, and is not.

Incrementality is a platform-wide solution that helps to measure the impact/effectiveness of an entire platform.

Incrementality is not a solution to measure the relative effectiveness of specific creatives. A/B split testing is a more appropriate tool for testing specific creatives.

Similarly, Incrementality is not a solution to measure the relative effectiveness of campaigns with similar objectives. For example, if two campaigns share the same objective — e.g., to acquire new customers — and users are exposed to both campaigns, these two campaigns must share the same group of Would’ves, and Incrementality would measure the combined impact of both campaigns. If, however, they don’t share Would’ves, the campaigns would contaminate each other’s “clean rooms.” A/B split testing by campaign is a more appropriate tool for measuring relative campaign effectiveness.

“Like all scientific experiments, Incrementality studies require time and data to produce concrete results.”

Verify the definition of the Would’ves.

The Would’ves are truly identical to users who were exposed to a brand’s ad. Make sure the point of determining if a user is/isn’t a Would’ve occurs after:

– The platform determines if the user is/isn’t in an audience inclusion/exclusion list

– The platform determines if the user is/isn’t part of the brand’s target audience

– The brand bids on the user

– The brand wins the auction to show an ad to the user

Only then can a user be deemed a Would’ve. If any of the above conditions aren’t met, the Incrementality solution may introduce bias into the Would’ves (the Control group) and materially impact the accuracy of the Control group.

And don’t forget: once a Would’ve, always a Would’ve. In statistical terms, make sure that there’s no bleed between treatment and control groups.

Ask for the details.

Any good incrementality solution should not be a black box, where you take the results at blind faith. Ask for full documentation of the platform’s Incrementality methodology, and get the details behind which conversions were test vs. control (Would’ves). Note, however, that privacy regulations may limit the amount or type of data that digital marketing platforms can share with you.