In this article, I outline a framework that helps prioritize development of programmatically driven notifications (applicable to mobile and web push notifications, email, SMS and potentially channels). The RRF framework is based on my experience — and mistakes made — building and scaling an activity notification service at SoundCloud and from subsequent lessons learned while consulting clients at Phiture.
The RRF framework will help you:
- Design and build the most impactful notifications first: increase user engagement and retention fast
- Avoid (or de-prioritize) building notifications that won’t drive growth
- Frame discussion and ideas about new notifications around a conceptual model based on impact
Introduction
Few initiatives have the potential to turbo-charge user engagement with your app as much as a well-designed notification system. Notifications let users know about events, interactions and developments that are worthy of opening up the app to engage with. They can be delivered over various channels; while they are most commonly associated with Push, notifications are also routinely deployed via email, in-app activity feed and sometimes even SMS, often giving the user the option to select their preferred way of receiving them.
The RRF framework is for prioritizing design and development of real-time activity notifications. The framework considers three dimensions (Reach, Relevance and Frequency) that, combined, contribute to the impact of push notifications.
Impact = Reach x Relevance x Frequency
A note about Impact
When I talk about impact in this article, I am referring primarily to increased user engagement (and, as a result of this, retention). While it’s true that some notifications (especially those featuring products available for purchase) may also drive increased monetization, the vast majority of notifications drive impact in the Engagement & Retention layer of the Mobile Growth Stack.
Activity Notifications drive impact in the Engagement & Retention Layer of the Mobile Growth Stack
With great power comes great responsibility…
Notifications can be an incredibly powerful tool, but like most powerful things, they also have the capacity to wreak havoc; few mistakes can be as costly to user retention as inadvertently flooding a user’s inbox or notifications tray with a deluge of irrelevant, erroneous or simply annoying messages.
Experiments aimed at increasing frequency of notification sends, or adding new types of notifications are often resisted by engineering teams fearful of betraying users’ trust by being ‘too spammy’. While this sentiment comes from a good place (a sincere desire to act in the best interests of the user and deliver a positive experience), it’s based on a hypothesis. More precisely, there are two conflated hypotheses: (1) that users inherently dislike notifications and (2) they will only tolerate a certain amount of them before abandoning the product in disgust.
It’s my experience that this view — while prevalent — is a gross simplification that risks leaving serious engagement upside on the table. It can also be highly accurate; it’s all down to the design of the notifications themselves.
The answer, as always, is to let the data lead the discussion: a notification system should be designed in such a way as to enable analysis not just of leading conversion metrics such as CTRs and short-term engagement, but also measure longer term engagement, retention and negative signals such as push opt-outs, email unsubscribes and app de-installations (all of which are measurable, but are often not monitored) versus a control group that doesn’t receive notifications.
Impact-driven development: Deciding which notifications to build first
Let’s assume that the team is aligned on building and testing at least some notifications, how should a team go about deciding which ones to build first, when faced with a backlog of interesting ideas? Ideally these decisions should be made with impact in mind, rather than going with the gut feel of a PM/Designer, or testing every idea.
In an effort to advance the discussion beyond a simple tug-of-war between viewpoints, I came up with a simple model that can help with the prioritization of notifications: the RRF framework.
Transactional notifications are part of the core product. Activity Notifications drive engagement with the core product.
Core product features that use notification channels (email, push, etc) to deliver essential functionality such as payment confirmations or password reset emails may be essential to the product, but are not so interesting from a growth perspective, as they present little upside for growth through further optimization.
The RRF model is designed to help product & growth teams prioritize the development of growth-centric programmatic notifications. Growth-centric notifications are designed to increase engagement with the core product and are typically built or significantly extended only once basic product-market fit has been established.
Different kinds of notifications may be best handled by different teams, as they serve different purposes
The RRF Framework aids prioritization when building programmatically-generated notifications that fall into the ‘Activity’ or ‘CronJobs’ rows in the above table. However, many of the concepts outlined in this document are worth bearing in mind when prioritizing lifecycle campaigns too. Lifecycle campaigns are behaviourally-targeted and are often created by marketing teams using visual campaign dashboards with marketing automation tools such as Braze, Clevertap, LeanPlum, etc.
The RRF Framework Explained
RRF is an equation for estimating impact of notifications:
Impact = Reach x Relevance x Frequency
Reach
It’s important to understand how many users will potentially qualify for a specific notification type: i.e. the approximate percentage of the total user base that will be notified when a certain event happens. This makes a huge difference (often several orders of magnitude) to the scale of impact that can be achieved with a particular type of notification.
The first question to ask is whether the notification is a one-to-one or one-to-many mapping.
Consider the example of a ‘new message’ notification in a social app, where a user sends a message to a friend or a private group of friends. This event will generate one notification per recipient. Even though the notification is highly relevant and personal, the reach is very low compared to the entire user base.
In the above example, a message to a single friend generates a one-to-one mapping: each message of this type will generate precisely one notification, with a reach of 1. If the message is sent to a group, the reach will equal the number of users in the group: in the case of a messaging app, this is often still in single digits, or low double-digits.
Overall send volume will, of course, also be a function of how many messages are being sent (frequency); WhatsApp manages to generate plenty of high-relevancy notifications from one-to-one mappings, for example.
Within the one-to-many classification, there are different orders of magnitude:
Illustration of the different levels of scale: not all events are equal when it comes to their potential to generate notifications.
The most impactful notifications have the potential to reach the entire user base (ideally including new users, who are particularly prone to churn). They have the highest potential for impact and should be built before creating lower-reach notifications for highly-engaged users.
The term ‘potential’ is used since the actual reach of each notification may well vary depending on the number of users who qualify to receive it. A single ‘new post’ notification on Instagram has the potential to reach the entire user base, but only if every user is following the user who posted. This is OK, since the relevance and frequency of these notifications is high and still classifies as a high-reach notification.
The highest reach of all would be a ‘broadcast’ notification such as an announcement of a new feature of short term special offer. While such notifications enjoy the highest possible reach, they usually score lower on relevance and hence should not be sent at high frequency.
‘Northern Lights’ Notifications have lower reach
Notifications that only kick in once a user is at a certain level of engagement may be more relevant and interesting, but won’t reach all of the users who churned before hitting that engagement milestone. I term these ‘Northern Lights’ notifications because they are only exposed to a relatively small percentage of the user base. They are generated only when a user has interacted with certain features/screens, explicitly subscribed to notifications about specific events, or is above a certain content consumption threshold (such that smart content recommendations are possible). They have a reduced reach that corresponds with the percentage of users who achieve the necessary level of engagement to qualify for them.
This isn’t an argument against building such notifications for engaged users; they have the potential to keep users interested and drive even higher levels of engagement with the most engaged segment of users, who are likely to have the highest monetization potential. However, they should be included only as part of a mix including full-reach notifications. When prioritizing which notifications to build first, go for those with total or near-total reach, since they typically drive the most overall impact, even when open rates (a reasonable proxy for relevance) are considerably lower.
The Reach and Frequency of Notifications dictate their potential for impact
Optimizing for Reach
Try to find things to notify users about that can be sent to the largest proportion of the overall user base. One-to-one messages, ‘Northern Lights’ notifications and those triggered by interaction with seldom-used features have low reach and hence will not drive high impact. Even when low-reach notifications are sent — and consumed — at high frequency, the lack of reach limits their overall impact dramatically.
Relevance
The more relevant and interesting the notification is, the higher the likelihood that the user will engage with it positively and the greater the desired impact. Users’ tolerance for frequency of notifications is directly proportional to the relevancy of the notifications; if notifications are relevant enough, the volume of notifications can be increased without negative effects (e.g. opt-outs, complaints or churn). Warning: The converse also applies!
Relevance can be evaluated once notifications are live by reviewing click/open rates, but during design phase, it can be hard to estimate the relevance of a notification. As a general rule, the more personalized and timely the content, the higher the relevance.
Let’s look at some examples:
Social interactions and ego-feeding notifications have high CTRs due to their direct personal relevance to the recipient
Take these ‘ballpark’ CTRs as a very rough guideline only: actual CTRs will vary by app category, size of user base and many other factors. Also, as Brian Balfour points out, growth benchmarks are (mostly) useless 😉 The benchmarks above are given primarily to demonstrate the difference in scale of impact between the notification types: this pattern holds true regardless of the app.
Optimizing for Relevance
Ensure that notification content is tailored to the explicit or implicit signals that you have from the user:
Personalization is key:
- Deliver personalized recommendations based on user tastes or behavioral history, social signals from other users interacting with the user or content they have posted.
- Other variables that can be used for personalization include local weather, geographic location, etc. Be creative and test things out.
- Delivering the notification at a time when the user is most likely to be awake is a basic form of personalization. Taking it one step further, it may be possible to deliver some notifications at a time when the user is most likely to be engaged (based on their individual usage history).
- Ensure notifications are localized to the user’s language (use their phone’s system language, or their explicit language preference rather than making an assumption based on their country).
Consider the timeliness of the message:
- Some events are incredibly relevant, but only for a short period of time: traffic alerts, flight delays and flash sales are examples of notifications where relevance decays sharply after the initial event occurs. Such events should generate notifications immediately, or not at all, to ensure high relevance.
Frequency
The events that can be used to trigger notifications occur with different frequency: a platform that has millions of users posting content has a very high frequency of ‘new post’ events, for example, each of which could be turned into interesting notifications for some portion of the user base. Events that occur with high frequency can generate many millions more notifications than those that occur less often, since each event might be sent to hundreds, thousands or even millions of specific users who qualify for a notification about it.
Of course, just because an event happens very frequently and could be used to generate millions of new notifications, doesn’t mean that users will appreciate them at that frequency. This is something to be discovered through careful testing. Nevertheless, notifications based off high-frequency events are fundamentally desirable, since they have high potential for impact if they do turn out to be engaging, there is enough ‘supply’ in the form of raw events to drive very high impact.
Optimizing for Frequency
Often, frequency is dictated by external events within the system (e.g. another user performs an action, which triggers an event that could be relevant for other users) or the real world (e.g. a band announces European tour dates and tickets go on sale). For the former, frequency of these events depends on overall activity within the platform, which is proportional to overall user activity. For the latter, frequency is largely outside of the control of the team (though could be increased through longer-term growth tactics such as increasing the number of supply-side partnerships or data integrations with other platforms).
As such, increasing frequency is often outside the direct control of a notifications team; a weather app cannot make it rain more often, ticketing platforms can’t force bands to tour more often, and so on. Notable — and important — exceptions to this rule are regular system-generated CronJob type events which trigger generation of weekly summaries, daily content recommendations, or other regularly-occurring events. Such events provide a fantastic opportunity to increase frequency: something that should be enthusiastically tested once relevance is established.
Using RRF to prioritize development
Given the RRF framework states that, for any specific notification type, Impact = Reach x Relevance x Frequency, it should be clear that the most impactful notifications score highly on each dimension.
Relevance is something that should always be optimized to its highest potential, since any increase in relevance will be beneficial in terms of impact. It can be proxied in live notifications by the average click-rate, though it may also make sense to look at downstream conversion factors (such as purchases or engagement within a given window after the notification was sent).
Relevance can be somewhat tough to predict in the design stage, though it’s worth trying to get an early indication by testing some example notifications with test segments of real users by sending them example notifications (or several, to test frequency) that appear to be programmatically-triggered but actually sent manually. CTR data from such test campaigns will help to gauge the potential relevance of the final notification, but watch out for the ‘halo effect’ whereby CTRs are inflated due to novelty factor.
Once a notification is live, relevance can often be increased (within its potential range) through iteration in the form of algorithmic improvements, leveraging additional data/signals from the user, A/B testing copy, etc.
However, bear in mind that optimization for relevance rarely changes the fundamental order of magnitude of impact that can be expected.
Let’s consider two extreme cases to underline this point:
Notification A: a direct message (one-to-one notification. Reach = 1) with 100% relevance (i.e. 100% CTR)
Notification B: a daily alert (daily frequency = 1) that is sent to the entire user base of 10m active users, with 0.5% CTR.
In this example, Notification B drives 50,000 clicks (re-engagements with the app) every single day. It also drives engagement with new users during the first crucial early days, and has a chance to re-activate lapsed users.
In order to exceed this impact, Notification A would need to be generated at least 50,000 times per day, and it would likely only reach engaged users who are already active on the platform.
Notification B works very well for Messaging Apps like WhatsApp and other highly-social platforms (ones with strong network effects and an emphasis on user-to-user communication), but in most categories of app, B would deliver impact at several orders of magnitude less than A.
Given that a notification’s potential will be mostly limited by Reach and Frequency, it’s helpful to consider these two dimensions when deciding which notifications to build first, from a backlog of ideas.
Real World Examples
Stars (high reach, high frequency)
Medium sends out highly relevant ‘new content’ notifications to all active users several times per day
Medium understands the value of high-reach combined with high-frequency. They send me at least two ‘new post’ push notifications per day, triggered by new content being published and personalized based on my reading history (and, likely, social signals) to increase relevance: a great example of a ‘star’ notification.
Goal for Star notifications: Continuously strive to increase relevance through increased personalization. Consider refreshing and testing new copy regularly to keep them interesting to users over the long term.
Northern Lights (low reach, high frequency)
Named after the enchanting visual phenomenon observable only far up in the northern hemisphere (low geographic reach), but frequently seen in these regions, Northern Lights are notifications that are typically seen more often by power users.
Facebook Page Admin notifications are only seen by power users who create their own Pages
Goal for Northern Lights notifications: Consider ways to increase reach without harming relevance. One way to achieve this is to use implicit rather than explicit signals to qualify users for notifications (make them opt-out rather than opt-in).
Full Moons (high reach, low frequency)
SoundCloud’s New Playlist notification occurs at low frequency: Playlists are created less often than new tracks are uploaded
Everyone sees a Full Moon now and again in real life. In the Notifications world, these are occasional notifications that most users receive with fairly low frequency, because the events that trigger them do not happen that often (at least for the majority of users).
The ‘new playlist’ notification from SoundCloud is high(ish) reach (all followers of a particular artist will qualify for the notification when the artist creates a playlist), but playlist creation happens relatively infrequently compared to some other events (such as new tracks being uploaded).
Frequency will grow over time if SoundCloud increases playlist creation on the platform, e.g. by making this feature more prominent in the platform, or encouraging casual listeners and curators to make playlists.
Goal for Full Moon notifications: Seek to turn these into Stars by finding ways to increase frequency (e.g. new triggers) without harming relevance.
Eclipses (low reach, low frequency)
Flight times change rarely, but it’s highly valuable to be notified when they do
This example from the WorldMate Android app is only sent occasionally (low frequency) and only to affected travelers who also happen to be WorldMate users (very low reach: often one-to-one), but scores incredibly high for relevance: no user would dream of de-installing the app for sending this notification. Note that this is also a prime example of a notification that would benefit from being sent via SMS (delivered even if the user has no data coverage).
Goal for Eclipse notifications: Build these later (lower priority), since they won’t drive the highest impact on growth metrics.
Take a Portfolio Approach (but make sure it includes a Star)
A great, growth-optimized mobile product will likely feature a mix of notification types from this matrix, which will be augmented and improved over time. Implement too many high-reach, high-frequency ‘Stars’ and users may start opting out or de-installing due to notification fatigue; this is why it’s good to pepper the notifications mix with some more occasional (but perhaps more relevant) ‘Eclipse’ notifications and to reward the most-engaged segment with ‘Northern Lights’ power-user notifications.
Evolving existing notifications for higher impact
It is sometimes possible to move certain notifications between quadrants (i.e. turning Full Moons into Stars by identifying additional triggers for them (thus increasing their frequency), or increasing the frequency of Cron jobs (e.g. a weekly recommendation that can become a daily recommendation by improving the underlying recommendation algorithm or increasing the amount of content on the platform).
A fantastic case study of transitioning a notification between quadrants comes from the Facebook app. In the example below, Facebook have used previous interaction (like or comment) with a post as implicit permission to deliver notifications every time someone else leaves a comment on that post. This increases the reach and frequency of their ‘Person X commented on Y’ notification (which used to only be sent to the original poster, rather than all subsequent users who replied to the original post) massively versus requiring me to explicitly subscribe to updates on the thread. Facebook routinely defaults to the opt-out model rather than opt-in in order to increase frequency and reach, massively increasing engagement on the platform.
Facebook qualifies users for notifications based on previous post engagement
Summary
Like all models, RRF is an abstraction; it considers only average reach, relevance and frequency, whereas in practice it’s possible (and advisable) to dissect notification performance at more granular level as some will exhibit significant variance.
Important notifications topics not covered by this article include:
- Volatility / frequency distribution (handling this is important for effective scaling)
- Aggregation (batching events into digest notifications).
- Best time to send notifications that are not highly time-sensitive (check out Andrew Chen’s analysis on this topic).
If you have comments, feedback or suggestions regarding the RRF Model, or have alternative ways of considering notification effectiveness, please leave comments in the section below: I would love to hear how others are approaching prioritization of notifications development.
Table of Contents
[…] content (Note: for more information on the relevance of push notifications, please refer to the RRF Framework). In other words, we need users to feel motivated to check the Notification Center for […]