How to Conduct Cohort Analysis for Your Mobile App

Data can be tricky to manage, and it doesn’t always show you the whole picture. Fortunately, there are ways to gain a deeper insight into your mobile app’s user behaviour. And cohort analysis can do exactly that. 

Imagine you have a recipe app, with data showing that vegetarian recipes were averagely popular. You’d probably try to feature them on the front page of your app, occasionally include them in customer emails, or even design your ad creative with some vegetarian imagery in mind.  All of this in the hope of it positively impacts your app’s engagement and retention. 

But diving deeper, what if you discovered that 100% of the users favouriting vegetarian recipes were male? This could change everything. You could use targeting of emails, push notifications, and advertising to put vegetarian food front and centre for males, providing females with a totally different experience. You could even consider spinning off a vegetarian-only app solely targeting a male audience. Your engagement could skyrocket. 

Welcome to the world of cohort analysis.

What is cohort analysis?

Cohorts are simply groups of people who share a common characteristic, such as time of registration, geographic location or age. Cohort analysis takes these groups and uses them as a lens to analyse your user data, allowing you to compare one group to another. 

But.

(There’s always a but.)

While cohort analysis is an incredibly useful tool in showing you where to look, it alone cannot provide a definite solution.

For instance, it will highlight the user data such as:

Males under 40, users acquired from Facebook ads, and users who downloaded the app six months ago have a higher engagement rate.  

But it’s not going to tell you why.

Instead, cohort analysis should be thought of as a tool to reveal where you should then apply your resources e.g. 

• spend more $ on targeting men under 40
• roll out the same Facebook ad creative to other platforms
• investigate how the app has changed in the last six months. 

You can think about it as the metal detector you use in the hunt for optimisation gold. 

The examples of cohorts

There are different types of cohorts you can analyse depending on the issues your app is facing. Here are some of the most commonly used in app marketing: 

• Demographic cohort – a group of users that share common characteristics based on demographics, such as age, gender, income, etc. For example, you can create a ‘UK users’ cohort and a ‘US users’ cohort to examine if there is any difference in engagement. 

• Acquisition cohort – a group of users who were acquired during a specific period. For example, you could compare users acquired through Google Ads and users acquired through Facebook Ads during the same time period to see if one has a higher lifetime value than the other (if you are collecting the right down-funnel data!). 

• Behavioural cohort  – refers to a group of users who exhibit similar behaviour within a specific time frame. For example, you could compare a group of users that joined before your new onboarding flow went live (60 days) and a group who joined after (30 days) to see if there was any difference in onboarding completion rates.

How to conduct cohort analysis

1. Identify the problem

As with any analysis, it’s imperative that you start from, well, the end. As in, you know what you want your end result to be – i.e. “we want to improve our retention rate” – before diving into comparing various groups. 

2. Formulate a hypothesis

The activity might have already been completed – you could have already changed your onboarding flow to improve your retention rate, for instance.  If that’s the case, you can use cohort analysis to compare the user conversion rate before and after the implementation of the flow. This would help you evaluate the effectiveness of the changes you’ve made and identify any further opportunities for improvement. 

Alternatively, you might be investigating, say, your most valuable users. In which case you can start with a hypothesis such as “males in their 40s have a higher lifetime value (LTV) for my fitness app”. 

3. Agree on metrics

What you track depends on your hypothesis. You may want to measure LTV, to discover the most valuable cohort of users; engagement to measure the user experience or conversion rate if you’re looking at paywall performance. Remember to agree on the time period you want to track it over. 

4. Define your cohorts

The group you choose will depend on your hypothesis. For example, if you’re examining the fitness app, you may want to define your cohorts by age. On the other hand, if you’re investigating the effectiveness of your new pricing model, you need to compare the group of users that signed up before it was implemented with those who signed up after, rather than comparing male vs female sign-ups over the past six months. 

5. Run your report

Back in the day, this might have involved a fairly weighty spreadsheet, but now there are plenty of tools with cohort analysis features. You may use for example Mixpanel, Amplitude, FullStory and even Google Analytics (to a certain extent). 

6. Analyse the results

Go back to your hypothesis to see whether your results support it. If they do, you can use that information to inform your next step – as we’ve already established, cohort analysis provides the information on performance, not necessarily the answer. 

So it might show you that your retention rate was higher after changing your onboarding – easy, the new onboarding stays. Or alternatively, it might show you that it’s actually women in their 30s with the highest LTV, in which case you may want to allocate more budget towards targeting women in your paid user acquisition campaigns. Information you gather can be also used to influence your product roadmap (you may want to redesign the app experience to appeal to that market) or your CRM platform (to design email and push campaigns to increase retention).

Analysing cohort data 

Here’s an example  of cohort data and analysis, illustrating the Click-through rate on the “Subscribe” button:

The above graph highlights the effect that optimising the paywall screen had on three different user cohorts. The cohorts were grouped based on their frequency of app visits per month. 

From the graph, we can see that the first paywall optimisation activity in month 3, had a positive effect on all three cohorts with an increase in the subscription click-through rate of between 10-14%. The effects of this paywall update began to diminish by month 5, indicating that a second round of optimisations was needed. Conversion rates can plateau over time so if you don’t make any changes to the design, users can become accustomed to the typical format. Therefore it is important to regularly modify designs.

The paywall optimisations made during month 6 also brought about an increase in the subscription click-through rate across all cohorts, ranging between 6-10%. The cohort group that experienced the greatest increase in the click-through rate between the optimisations was the ‘occasionals’ cohort – those that visit the app between 2-6 times per month. After identifying this cohort as having the highest click-through rate, we can dive deeper into the user behaviour to understand their steps to this point, in order to encourage similar user groups to convert.

Key takeaways: 

• First paywall optimisation increased the subscription rate by 10-14% for all three cohorts
• The effects of this optimisation plateau around month 5, so additional optimisation was needed
• Second paywall optimisation increased the subscription rate by 6-10%, impacting “occasionals” cohort the most
• We used our findings to understand how we can encourage similar user groups to convert

Using cohort analysis holistically

Cohort analysis traditionally sat in the Product team, but increasingly, it’s being seen as something that can be used across the business, thanks to the power it holds to deep dive into user data. Here’s how you can use cohort analysis across other teams.

Using cohort analysis for User Acquisition

It’s generally accepted that organic users have a higher LTV than paid acquisition users, due to the level of intent. However, lumping all user acquisition channels into one can be incredibly misleading. 

Cohort analysis can support your paid user acquisition efforts as it aims to understand different types of audiences and the reasons why they convert or not. Let’s say you’re running paid campaigns on Facebook and Google at the same time. By separating and comparing both channels as cohorts, you can get a better understanding of how users behave week on week. This allows you to see which channels are most effective at driving high-quality traffic or identify low-quality traffic sources. It can also support UA teams in their targeting efforts. Based on the findings that the cohorts provide, they can create look-a-like audiences, use demographic and location data to create better audiences or even adjust their event targeting. 

Using cohort analysis for Product 

One simple approach to analysing user behaviour is to group them by their operating system, as this can shed light on where (iOS & Android),  your product team should prioritise their efforts. For example, if the conversion rate is significantly lower on one operating system compared to the other, it could indicate a potential bug in the purchase process or that many of the downloads on one operating system are generated by bots. 

It’s most often used, however, to simply check if an update has had the desired effect. For example, do your recent users subscribe more frequently now you’ve made that change to the checkout process? Or has your new branding turned users off or on? Cohort analysis allows you to investigate those scenarios.

Using cohort analysis for CRM 

Cohort analysis is a great way of making sure that you’re avoiding common drop-off points, nurturing your users through to the other side. By grouping your users by lifecycle stage, you might find that month two is where you see the most uninstalls, and then put something in place to re-engage these at-risk users, reducing churn.

Summary

• Cohort analysis allows you to dive deep into your users and is a fantastic tool for uncovering surprising insights regarding your users.
• It won’t necessarily provide you with information on what you need to adjust to change your key metrics, but it will show you where you should focus your efforts.
• Cohort analysis insights should be used and shared across the business, as other teams may benefit from the data they provide.