Mixpanel - Analytics for startups

Update: ETA is within the next day or day and a half to be fully caught up.

Update (Feb 4th, 2:50 AM): Seeing good stability across new infrastructure and we're crunching data a lot faster.

Update (Feb 4th, 7:35 PM): Processing through the backlog of data. It's about 1 day backed up however you should see data before Feb 3rd 17:00:00 UTC at least.

Update (Feb 6th, 11:56 PM): We are now back to real-time and all systems are stable.

For the past few days many of our customers must be wondering what's been going on with Mixpanel and why they have been seeing a lot of red percentages and zeroes. The truth is a combination of factors stemming from simply trying to scale to the large volume data that we process in real-time--it's never been easy and at each step we have to get innovative and diagnose problems. The good news is, we've been getting better at this as time goes.

We diagnosed our problems about 1.5 months and had a plan of action to scale parts our system out again in the smoothest way possible. The goal was not to just scale out the system and augment performance by a huge order of magnitude but it was to do so in a manner in which our customers wouldn't even notice. We certainly failed at the latter goal. The problem was that we were forced to roll out the new system sooner than expected to handle scale which didn't provide enough time to thoroughly vet some of the systems and issues we have been able to work through under less stress. 

If we've lost you as a customer or you're uneasy with doing your analytics with us I would ask that you reconsider. When things like this happen you can be sure we're absolutely getting the least amount of sleep physically (sometimes none) possible till we fix the problems that hurt our reputation as a company. Also, we've reached a huge milestone with the new infrastructure we've rolled out to the point where we can focus on creating a reliable service capable of handling an extremely large data volume in real-time. In our opinion, things will only get better at this point and we're the kind of startup that tends to rise to the challenge no matter what.

If we've learned anything from this process we learned to be much more proactive and providing lots of status updates about what's going on since we know you rely on us for your business day to day. We're sorry for not doing that better. If there's anything else you wish to add, please leave us a comment or email us at support@mixpanel.com.

To address any of your concerns right away:

1. There was no data loss.
2. We're working through a backlog of data that could take an additional 1-1.5 days before we get to-the-second real-time again.
3. Retention tables/charts have been reset--this was a planned side effect not a result from problems we've been having.
4. You should see new data in your report pages being crunched over the past days or hours that have been missing constantly.

We sincerely apologize for the problems we've had over the past days and we'll be working on ways to prevent things like that from happening again.

Loading mentions Retweet

Due to an edge case programming error you may notice your unique visitor traffic is spiking if you do a high data volume with us. We regret to say that we made a mistake and you should ignore it. Please note this pertains to only the average and unique visitor traffic for events and properties not funnels or retention.

Rest assured we fixed the problem and looked for additional places where the same bug could also be. Taking how we run things here one step further, at Mixpanel we heavily are now using unit tests to make sure data is precisely 100% accurate all the time. Any change made that could effect how your data is computed and analyzed will be run against our unit tests to ensure we are always accurate.

As a startup we'll make mistakes but we're working towards increased stability both in our software and infrastructure. Things will only get better from this point forward.

If you have a question, please email us at support@mixpanel.com and we'll happily answer.

Loading mentions Retweet

The guys over at Backtype wrote a post about their adventures in split testing. Check it out:

http://tech.backtype.com/split-testing-with-mixpanel

Loading mentions Retweet

We would like to inform everyone that we've updated our Javascript client library in favor of a new architectural design that became more necessary. The primary change is that we need to change the way developers use the actual mpmetrics object, many customers of ours also use widgets provided by companies like Uservoice.com and because of the current implementation and due to the fact that Uservoice is a customer of ours, the mpmetrics object gets overriden and cuts off your metrics.

The change is simple, we now require developers to instantiate a brand-new object called MixpanelLib which you can set to a variable called mpmetrics:

<script type="text/javascript">
try { var mpmetrics = new MixpanelLib("YOUR_TOKEN"); } catch(err) {}
</script>

You can download or source the new Javascript library at: http://api.mixpanel.com/site_media/js/api/mixpanel.js

That's it, really simple. You can continue to call mpmetrics like you always have. We're working with Uservoice to rename their variable mpmetrics to something else so it does not conflict. This allows you to continue to inline our javascript without overriding some one elses mpmetrics object and in the future allows you to use multiple projects to log data to much more easily as well.

If you have any questions or concerns, please email us at support@mixpanel.com.

Loading mentions Retweet

Here's the idea: sometimes you want to optimize for engagement, rather than something more concrete like revenue or conversion rate.  Measuring engagement is a trickier task, (one you should use Mixpanel for), but that just makes it more interesting.

This graph may be a bit difficult to interpret. It shows the frequency of visitors with different comment counts - for example, there might be 1,000 visitors who left 2 comments, 345 who left 3 comments, and so on.  This means that a point (X, Y) on the curve tells you that there were Y visitors who left X comments.  Because this is just an example, the specific numbers don't really matter.  The most important part is the overall shape of the curve.

We can see that most users don't comment at all, and that there are very different behaviors between groups.  The control group (green line) has more users that write a small number of comments each, while the experimental group (red line) has fewer active users who comment frequently.

Loading mentions Retweet

There are a lot of articles on the internet related on how to A/B test and there are numerous open source tools that are available to do it but nobody does a great job doing macro level A/B testing which we think is extremely important.

I personally learned about this idea from Eric Ries when he talked about a case where IMVU A/B tested the text of something only to find it was able to increase the conversions of something like revenue and they had no idea it had this effect on their users. It goes to show that an A/B test may increase conversions of something desirable but adversely effect something else or do completely the opposite of that scenario which would be profound.

When we designed Mixpanel, we thought about this idea and made sure it would be feasible to do this in the future so I'd like to show you how we're able to do this for you today, you can literally create an account and try it yourself or follow along in some manner. We think being able to do these macro level A/B tests really differentiates ourself from current tools available today.

Basic understanding

With Mixpanel, you're always logging events with a track() function:

mpmetrics.track("played a song", {"genre" : "hip-hop"});
mpmetrics.track("added to playlist", {"genre" : "pop"});

By Mixpanel standards, this is pretty basic API usage, this lets our customers track when a user plays a song or adds a song to their playlist but we take it one step further by letting them segment those events by the genre of that song which has its own merits in terms analysis.

Regardless, understanding properties ({"genre" : "pop"}) is the cornerstone in your understanding in how we can macro-level A/B test. These are user-defined and you can add more than one.

Creating the test

Any A/B testing article will generally explain that you can split users using the modulo operator on some user id:

function pick_background_color(user_id) {
    var test_id = user_id % 2;

    if (test_id) {
        return "red";
    } else {
        return "blue";
    }
}

This is basic code that chooses the background color to use based on a user_id provided to us so that the user always sees the same background color consistently. You should be able to setup a basic test to do almost anything you want just using the modulo operator and some basic programming logic.

Logging the data

Once you have the test created, what you can then use Mixpanel's properties as a way to log the test data. For example, lets say we have a music site and want to see how a red versus blue background effects our users behavior on the website.

So what we do, when we decide the background color we'll use in the CSS using the pick_background_color() function. In that function, we'll modifiy and add some Mixpanel code:

function pick_background_color(user_id) {
    var test_id = user_id % 2;

    if (test_id) {
        var color = "red";
    } else {
        var color = "blue";
    }

    mpmetrics.register({"test-background" : color});

    return color;
}

What mpmetrics.register does is it binds that property to every event along with any other events also bound to that event. This allows us to see how a red vs. blue split in background color effect every event on our music site. So we'll be able to see whether a blue or red background increases overall engagement: songs being played and songs added to a playlist (or all the other events being logged).

That's all it takes extra: mpmetrics.register();

On our reports, you can click on the "View" button and then click properties to get to your "test-background" property to see its effect and the split of red vs blue for that specific event.

Wrap up

It's not enough to just test whether a blue background color increases song plays, you might find it decreases songs being added to playlists--this is important to know because you may reconsider your product decisions if you're a site like Muxtape who thrived off users creating new playlists that would be shared around increasing their overall traffic.

Imagine if these metrics were related to your bottom line, you better know how certain things effect everything else otherwise your product could be headed down an unexpected and undesirable path.

Loading mentions Retweet