See the transcript on incrementality testing below:
What is incrementality testing and how does it work?
Incrementality measurement, otherwise known as lift testing or lift modelling, offers a scientific approach to measuring ad efficacy on top of organic or existing user engagement. Incrementality aims to answer the question of how valuable your campaigns really are.
It helps to answer the question: Would these users have gone on to complete this action that you’re aiming to achieve anyway?
To start, you need to first define your audience. We usually recommend an audience size of around 50 to 100,000 users and you separate that audience into two groups. The first group is a test group which is served the ads as normal, and the second group is a control group which is held back and served no ads. It is recommended that the control group include at least 15% of your audience to ensure that the test holds statistical significance. The idea is you can then calculate the gap or the ratio between the measured KPI from both the control and the test groups to see the incremental lift and the real impact that your campaigns are having on your users.
How should you go about defining your goals?
The goal of incrementality testing is to understand the incremental lift between those that saw the ad and those that did not. In order to do that you need to formulate a question or a hypothesis that you need answered.
So for instance: Would showing a remarketing ad on Facebook to users after three days of inactivity increase my revenue?
The structure of that question clearly defines:
- Audience: users that have installed but not been active for three days
- Channel: which is on Facebook in this example
- Creative: which is a remarketing ad
- Goal: which is to increase your revenue
With that, you have everything you need to test incrementality.
How do attribution and incrementality testing fit in together?
Attribution is vital in understanding where to spend your marketing budget. The same goes for incrementality testing, however incrementality is really considered as the gold standard of measurement. Attribution is a great tool for quick and accurate decisions, whereas incrementality requires a little more resources, time and budget. Incrementality is not something that you can test continuously, you really need to have a timeframe, usually around 30 days, whereas attribution can give you a real-time understanding to determine the value of your marketing efforts, reduce cannibalization, and allocate your budget effectively.
Attribution and incrementality testing serve the same purpose but incrementality will help you understand those deeper metrics like incremental ROI and incremental CPA.
What is A/B testing and how does that fit in?
A/B testing is a very similar style of testing,it is an approach to effectively validate your incremental lift. The key difference is that there is a control group. incrementality is effectively an A/B test with that control group, it aims to achieve the same thing in terms of testing similar groups on their behaviour. Similarly within incrementality measurement, as long as you have that control group as your baseline you can A/B test on different networks, so you could split your audience, let’s say 25% on Facebook, 25% on Google, 25% on Tik Tok and the remaining 25% would be your control group, you would then be able to compare each network and the incremental lift to the control group but similarly between the two ad networks as well.
The challenges of incrementality testing
There are a few important requirements for incrementality testing which include; budget, audience size, etc. One of the challenges with incrementality would be ensuring accuracy of testing. If you’re working on a number of different ad networks, ensuring that the entirety of that test group is actually served the ads is a challenge, because of course we can’t guarantee 100% that the test group will all be served those ads.
The key challenge right now in the wake of iOS 14 is working towards an IDFA list solution. Audience building traditionally is dependent on device IDs (unique identifiers). As we shift towards a privacy centric world AppsFlyer is working on a tool that works on a geo split, so geo based audience building, as opposed to being reliant on uniquely identifying users within that audience.
If you want to find out more about how improving the mobile customer experience can impact your app marketing campaign or you have any app marketing questions feel free to reach out to the Yodel Mobile Growth Team. Additionally, you can sign up to our newsletter for exclusive industry updates and app marketing insights from the ASO Agency of the Year. Want to find out more about optimising your app? Make sure to subscribe to our Mastering Mobile Marketing video series. Follow us on LinkedIn, chat with us on Twitter @yodelmobile, and join our #AppMarketingUK LinkedIn group.
