A/B testing to increase your conversions the Eric Ries way
PostedThere are a lot of articles on the internet related on how to A/B test and there are numerous open source tools that are available to do it but nobody does a great job doing macro level A/B testing which we think is extremely important.
I personally learned about this idea from Eric Ries when he talked about a case where IMVU A/B tested the text of something only to find it was able to increase the conversions of something like revenue and they had no idea it had this effect on their users. It goes to show that an A/B test may increase conversions of something desirable but adversely effect something else or do completely the opposite of that scenario which would be profound.
When we designed Mixpanel, we thought about this idea and made sure it would be feasible to do this in the future so I’d like to show you how we’re able to do this for you today, you can literally create an account and try it yourself or follow along in some manner. We think being able to do these macro level A/B tests really differentiates ourself from current tools available today.
Basic understanding
With Mixpanel, you’re always logging events with a track() function:
mpmetrics.track("played a song", {"genre" : "hip-hop"});
mpmetrics.track("added to playlist", {"genre" : "pop"});
Regardless, understanding properties ({“genre” : “pop”}) is the cornerstone in your understanding in how we can macro-level A/B test. These are user-defined and you can add more than one.
Creating the test
Any A/B testing article will generally explain that you can split users using the modulo operator on some user id:
function pick_background_color(user_id) {
var test_id = user_id % 2;
if (test_id) {
return "red";
} else {
return "blue";
}
}
This is basic code that chooses the background color to use based on a user_id provided to us so that the user always sees the same background color consistently. You should be able to setup a basic test to do almost anything you want just using the modulo operator and some basic programming logic.
Logging the data
Once you have the test created, what you can then use Mixpanel’s properties as a way to log the test data. For example, lets say we have a music site and want to see how a red versus blue background effects our users behavior on the website.
So what we do, when we decide the background color we’ll use in the CSS using the pick_background_color() function. In that function, we’ll modifiy and add some Mixpanel code:
function pick_background_color(user_id) {
var test_id = user_id % 2;
if (test_id) {
var color = "red";
} else {
var color = "blue";
}
mpmetrics.register({"test-background" : color});
return color;
}
What mpmetrics.register does is it binds that property to every event along with any other events also bound to that event. This allows us to see how a red vs. blue split in background color effect every event on our music site. So we’ll be able to see whether a blue or red background increases overall engagement: songs being played and songs added to a playlist (or all the other events being logged).
That’s all it takes extra: mpmetrics.register();
On our reports, you can click on the “View” button and then click properties to get to your “test-background” property to see its effect and the split of red vs blue for that specific event.
Wrap up
It’s not enough to just test whether a blue background color increases song plays, you might find it decreases songs being added to playlists–this is important to know because you may reconsider your product decisions if you’re a site like Muxtape who thrived off users creating new playlists that would be shared around increasing their overall traffic.
Imagine if these metrics were related to your bottom line, you better know how certain things effect everything else otherwise your product could be headed down an unexpected and undesirable path.
Any good A/B testing article should explain that while you *can* split users on userid, doing so limits your ability to run multiple A/B tests at once which is bad. Also you need to do some statistics to tell the difference between chance fluctuations and statistically significant results.See my slides at http://elem.com/~btilly/effective-ab-testing/ for a much better explanation of how to A/B test
good article.here’s another program to try for A/B testing.