Introducing A/B Testing for Notifications
Ever since Mixpanel introduced Notifications, our customers have used it to run A/B tests on their own marketing and outreach efforts. And why wouldn’t they? Given how easy it is to conduct targeted outreach via email, SMS, push, and in-app notifications, A/B testing is a natural fit. So natural, in fact, that we decided to make it even easier by building it right in.
With Mixpanel’s new A/B testing feature, you can send variations on a notification to see which one speaks best to your audience. As always, you can tie notifications to conversion events (or any event you please), which will instantaneously tell you how your users react to different approaches. Our analytics on A/B testing also save you time: predictive algorithms are built in to determine which variation is likely to have the highest success rate in the long run. Below is a walk-through of how exactly this will look in Mixpanel.
When you create a new notification, or go to update an existing notification, you’ll see an option to create a variant. The example here is for email, but A/B testing is also available for push, SMS, and in-app notifications. You can create as many variants as you like.
When you go to target your users, you’ll see a pop-up (above) that allows you to set the distribution of your different notification variants. For example, you may have one variant that’s really sassy; perhaps you’ll only want to send it to 5% of your customer base to see whether your customers respond positively. You can then track the progress of your notification variants and compare them.
The graph above shows how many notifications were sent for each variant. You can then see the conversion rate from each notification variant to any event that occurs in your app — this can be anything from a signup event to simply tracking the notification open rates. Think of it as a tiny conversion funnel, specific to this set of notifications.
The chart above displays all the stats for your notification and its variants. You can see how many of each variant were sent, the conversion rate for each, and their improvement over the original. The last column, “chance to beat original,” predicts the winner of the bunch — the notification variant that you’re likely best off using as the primary notification in future. Naturally, you can also clearly see the conversion rates for each variation over time (this is Mixpanel, after all).
That’s A/B Testing for Notifications! More information about general notification analytics can be found here. If you have any questions or feedback, please don’t hesitate to reach out to our Support Engineers at email@example.com.