Mixpanel - Analytics for startups

The guys over at Backtype wrote a post about their adventures in split testing. Check it out:

http://tech.backtype.com/split-testing-with-mixpanel

We would like to inform everyone that we've updated our Javascript client library in favor of a new architectural design that became more necessary. The primary change is that we need to change the way developers use the actual mpmetrics object, many customers of ours also use widgets provided by companies like Uservoice.com and because of the current implementation and due to the fact that Uservoice is a customer of ours, the mpmetrics object gets overriden and cuts off your metrics.

The change is simple, we now require developers to instantiate a brand-new object called MixpanelLib which you can set to a variable called mpmetrics:

<script type="text/javascript">
try { var mpmetrics = new MixpanelLib("YOUR_TOKEN"); } catch(err) {}
</script>

You can download or source the new Javascript library at: http://api.mixpanel.com/site_media/js/api/mixpanel.js

That's it, really simple. You can continue to call mpmetrics like you always have. We're working with Uservoice to rename their variable mpmetrics to something else so it does not conflict. This allows you to continue to inline our javascript without overriding some one elses mpmetrics object and in the future allows you to use multiple projects to log data to much more easily as well.

If you have any questions or concerns, please email us at support@mixpanel.com.

Here's the idea: sometimes you want to optimize for engagement, rather than something more concrete like revenue or conversion rate.  Measuring engagement is a trickier task, (one you should use Mixpanel for), but that just makes it more interesting.

This graph may be a bit difficult to interpret. It shows the frequency of visitors with different comment counts - for example, there might be 1,000 visitors who left 2 comments, 345 who left 3 comments, and so on.  This means that a point (X, Y) on the curve tells you that there were Y visitors who left X comments.  Because this is just an example, the specific numbers don't really matter.  The most important part is the overall shape of the curve.

We can see that most users don't comment at all, and that there are very different behaviors between groups.  The control group (green line) has more users that write a small number of comments each, while the experimental group (red line) has fewer active users who comment frequently.

There are a lot of articles on the internet related on how to A/B test and there are numerous open source tools that are available to do it but nobody does a great job doing macro level A/B testing which we think is extremely important.

I personally learned about this idea from Eric Ries when he talked about a case where IMVU A/B tested the text of something only to find it was able to increase the conversions of something like revenue and they had no idea it had this effect on their users. It goes to show that an A/B test may increase conversions of something desirable but adversely effect something else or do completely the opposite of that scenario which would be profound.

When we designed Mixpanel, we thought about this idea and made sure it would be feasible to do this in the future so I'd like to show you how we're able to do this for you today, you can literally create an account and try it yourself or follow along in some manner. We think being able to do these macro level A/B tests really differentiates ourself from current tools available today.

Basic understanding

With Mixpanel, you're always logging events with a track() function:

mpmetrics.track("played a song", {"genre" : "hip-hop"});
mpmetrics.track("added to playlist", {"genre" : "pop"});

By Mixpanel standards, this is pretty basic API usage, this lets our customers track when a user plays a song or adds a song to their playlist but we take it one step further by letting them segment those events by the genre of that song which has its own merits in terms analysis.

Regardless, understanding properties ({"genre" : "pop"}) is the cornerstone in your understanding in how we can macro-level A/B test. These are user-defined and you can add more than one.

Creating the test

Any A/B testing article will generally explain that you can split users using the modulo operator on some user id:

function pick_background_color(user_id) {
    var test_id = user_id % 2;

    if (test_id) {
        return "red";
    } else {
        return "blue";
    }
}

This is basic code that chooses the background color to use based on a user_id provided to us so that the user always sees the same background color consistently. You should be able to setup a basic test to do almost anything you want just using the modulo operator and some basic programming logic.

Logging the data

Once you have the test created, what you can then use Mixpanel's properties as a way to log the test data. For example, lets say we have a music site and want to see how a red versus blue background effects our users behavior on the website.

So what we do, when we decide the background color we'll use in the CSS using the pick_background_color() function. In that function, we'll modifiy and add some Mixpanel code:

function pick_background_color(user_id) {
    var test_id = user_id % 2;

    if (test_id) {
        var color = "red";
    } else {
        var color = "blue";
    }

    mpmetrics.register({"test-background" : color});

    return color;
}

What mpmetrics.register does is it binds that property to every event along with any other events also bound to that event. This allows us to see how a red vs. blue split in background color effect every event on our music site. So we'll be able to see whether a blue or red background increases overall engagement: songs being played and songs added to a playlist (or all the other events being logged).

That's all it takes extra: mpmetrics.register();

On our reports, you can click on the "View" button and then click properties to get to your "test-background" property to see its effect and the split of red vs blue for that specific event.

Wrap up

It's not enough to just test whether a blue background color increases song plays, you might find it decreases songs being added to playlists--this is important to know because you may reconsider your product decisions if you're a site like Muxtape who thrived off users creating new playlists that would be shared around increasing their overall traffic.

Imagine if these metrics were related to your bottom line, you better know how certain things effect everything else otherwise your product could be headed down an unexpected and undesirable path.

Last week Patrick McKenzie of http://www.bingocardcreator.com did a great writeup on using Mixpanel with Ruby on Rails - you should take a look!

Why You Should Use Analytics

The fundamental reason you want to use analytics is because it lets you know how your customers interact with your site and that this drives decisionmaking about your site design, future business directions, and the like.

You might say "Wait, why don't I just talk to customers?" You should talk to your customers, but customers:

• have more important things to do than talk to you
• are not experts about your site
• misremember things
• resent being tested on like rats in a twisted science experiment ("Would you hit this bar for cheese more if it was green, or blue?")

Analytics has none of these problems. The data you collect:

• is always there, waiting for your analysis
• is totally expert on use of your site
• is as accurate as your collection method
• has no issue with answering arbitrary, inane questions

Super Properties have been available for Mixpanel for a few weeks now, but not everyone knows that they exist or the true value super properties can provide for people who use Mixpanel.

Quick description

On Mixpanel, when you track an event, you are allowed to have properties attached to that event. So if a user is playing a song, you may want to know what type of song, maybe where that song was played from, and what artist so you can visualize and understand more about your users. Properties describe events.

There are properties that you want to attach to all events, such as gender. Super properties are just global properties you can set across all events so you do not have to tediously include them with everything you log. Super properties are available for our Javascript library and soon our Facebook version as well. For information, we encourage you to read the documentation we have available.

Case 1: Tracking groups

To make your life easier, we allow you to track global variables such as gender, a user type, logged/not logged in. It's really dead-simple:

mpmetrics.register({"gender": "male", "session" : "logged in"}, "events");

When you view your report you'll be able to see the segmentation of male vs female users that interact with all events across your entire website. This is very useful because you get to see how a group of users whether they are male/female, logged in/not logged in respond in certain ways and how they compare.

Tracking groups can be very helpful, you may want to just track how a certain group of users who potentially purchase gold interact with your application for instance versus those who do not. Super properties can help you realize that you have different types of users you may need to optimize for and think about while building new features.

Case 2: Experimentation

Super properties actually encourage experimentation across your application. Imagine for a moment that you want to test a white background against a black background. When you view the report pages, you'll be see how different color backgrounds effect engagement across your entire application, not just one specific event or type of conversion. It is extremely valuable to do macro level A/B testing rather than focus on micro-level characteristics (e.g. Does this one orange button increase conversions)

If you had a music website, you might be surprised to see that a different color background makes people play more songs but decreases the amount of revenue you make, while another color does the exact opposite. The end result helps make the right decision for what you're building.

Case 3: Origin tracking

When building your application it is extremely important to understand where users are coming from because that origin may be vital towards your growth. Knowing where your users are coming from is only half the battle, with super properties you can actually find out how a point of origin effects the overall experience of a user.

One of our most useful tools happens to be Funnel Analysis, funnel analysis allows you to see how users drop off at each step of your flow. With super properties, you could essentially register the origin at step 1 and see which origin converts the best at the end of a funnel.

As an example, imagine you are building a Facebook application. Facebook provides numerous ways to funnel in users: invites, notifications, news feed items, cross promotion/advertising but which one performs the best for your application? Which one should you focus on? Super Properties will be able to show you exactly that information.

When we pitch Mixpanel to people, a lot of them tell us that they're busy working on their core product, and that they're not at an optimization stage yet.  I'd like to point out how valuable analytics data can be for learning about your customers - something that is crucial to product development.

A couple of really valuable aspects are

1. No selection bias
2. Understanding feature use

   
Selection bias

Selection bias often occurs when companies ask customers for information.  The customers with the strongest opinions are most vocal, and skew the dataset.  This means that the data you get back from surveys may not actually be a valid representation of your customers.

A great thing about automated analytics is that there is no selection bias.  You track what EVERYONE is doing, so there's no way to get only a subset of opinions.  That means you can trust the analytics data you gather a bit more than the feedback you get from talking to users.

Feature use

As a startup, we spend a lot of time cranking out new features.  However, we can't afford to spend too much time working on something before we know it works.  This means we push out new stuff way before we're entirely happy with it - but it lets us start getting actionable feedback immediately.

This feedback comes in the form of data.  We can instantly see if the new feature is getting use or getting ignored.  If it's getting use, we see what aspects of it are most popular and we can focus development on those. 

If it's getting ignored, that is just as telling. A new feature that no one wants is useless, and we either lower its priority or scrap it entirely.

We heard a real-life example of this from one of our users the other day. Joey Liaw of Voxli (YC W09) is using Mixpanel, and he posted this to friendfeed:

Top three push to talk bindings for our users this week: 16% use Left Control, 11% use Middle Mouse Button, and 10% use Tilde. I put mouse support in because David A. asked for it, but I thought he'd be the only person using it. Glad I was wrong. Would be interesting to correlate this against what apps users run, or their Arena ranking.

He only knows this because he started tracking it.  It sounds obvious, but so many people are flying blind - they release new things and have no idea what happens next.

I'm sure you can think of times in the past when you added something to your product and never knew if it was getting any use - what were they?