Last week an experienced digital marketer visited our offices to learn about Mixpanel. His background was in web analytics, but he had recently been hired by a growing mobile company to professionalize their digital marketing.

When he arrived at his new company, he was shocked to learn they were paying thousands of dollars for users via a number of ad networks (iAd/MoPub/Admob/Tapjoy, etc) but didn’t have numbers showing how engaged the users acquired this way were. He had no way to know if a user acquired from one campaign brought in more revenue than one acquired a different way.

Source attribution on the web is relatively easy. If I want to bring users via AdWords to my site, I’ll make the landing page link from the ad something like this:

http://www.mysite.com/landingpage?utm_source=AdWords&utm_medium=PPC&utm_campaign=NameOfMyCampaign

Once I’ve done that, in Mixpanel the utm_source, utm_campaign, and utm_medium all become super properties I can use for segmentation. This is vital to determining how effective each campaign, medium, or source was in terms of conversions to actual users or revenue generated and retention. Without this information you cannot make an informed decision about how to allocate your marketing resources. This has become a standard practice for web-based marketing.

However, as I told the marketer I was speaking with, for mobile apps, tying a user’s behavior within the app with their acquisition source data is more complicated. This means very few mobile app marketers actually do it. However, it is possible! If you go through the trouble of linking up your ad network with Mixpanel you can hold your ad network accountable for the quality of the users they send you.

Android

Acquisition source attribution is much easier in the Android Marketplace than in the iOS App store. This is because if you provide a link to your application that looks like this:

http://market.android.com/details?id=your.package.name&referrer=your_referrer_parameter

Android will make the your_referrer_parameter available to your app when it first launches., This parameter needs to be a series of specific key / value pairs and be URL encoded. The keys you can use are utm_source, utm_medium, utm_term, utm_content and utm_name. utm_source, utm_medium and utm_name are required. You can use these values to categorize the campaign in different ways. A completed Android Market URL would look something like this:

http://market.android.com/details ?id=your.package.name&referrer=utm_source%3Dtapjoy%26utm_medium%3Dbanner%26utm_campaign%3Dbrainy%2520images


To access the referrer information, create a Broadcast Receiver in your app registered to the com.android.vending.INSTALL_REFERRER intent filter. A good code example for doing this is available here.

Once you have access to your referrer parameter, you can register it as a Mixpanel super property, so all future events will be associated with it.

iOS

Acquisition source attribution is much more difficult on iOS. Apple does not provide any capability to pass a referrer string through to your app from a link to the app store. The result is that when someone installs an app for the first time, you are blind as to where they came from.

To solve this problem, traditionally the ad networks would record the UDID (a unique identifier assigned by apple to every iDevice) of anyone who tapped on one of their ads. They also provide you with a code snippet that you add to your application. This code snippet will ping the ad network’s server when your app first opens with the UDID of the device that opened the app. The ad network will take credit for all UDIDs that both tapped on their ad, and launched your app for the first time.

Much has been written about Apple’s recent rejection of some apps that access UDIDs, causing a minor panic. Though it is important not to panic (apps that disclose they collect the UDID are not being rejected) it is clear that the industry is moving away from UDID to an array of alternatives. In general these alternatives are drop-in replacements for UDID, and don’t change this general paradigm.

Whether an ad network is using UDID or some other identifying string, the app itself does not know when it first launches that it was discovered through a certain acquisition source. This means you can’t just register a super property the way you would in Android to perform easy segmentation of user behavior in Mixpanel. Instead, you have to add the data to Mixpanel after the fact.

Most ad providers will provide you with a flat file that indicates which UDIDs (or other identifier) installed after tapping their ad. This is usually generated daily, and is accessible in an automated way via FTP or S3. To get this data into Mixpanel, write a script that processes this data file and uses our HTTP API to insert a back dated event for each user called “Annotated Signup.” At a minimum this event needs to have the following properties:

• The distinct_id of the user
• The time the user’s account was created
• The acquisition source of the user

The distinct_id is the identifier you are using to identify users within Mixpanel. Depending on your implementation this could be your own internal ID or the identifier Mixpanel uses (a one way hash of the MAC address).

The time should be a UTC unixtime stamp representing when the account was created. This is most likely recorded in your database.

The acquisition source is the name of the ad network or the campaign the user came from. This will be determined by which file you got the identifier from.

The trickiest part of this is mapping the identifier you get from your ad network to the distinct_id used within Mixpanel. You need to understand exactly which identifier system your ad network is using, and add logic to your application to send that identifier to your server, so you can store it in your database and associate it with each of your users. You can then use your database to lookup each identifier you get from the ad network, translate it to the distinct_id you are using for Mixpanel, and then send the event to Mixpanel.

This “Annotated Signup” event can be used as the first step for your funnel reports and retention reports, allowing you to understand how users from different sources convert to different actions and retain in your application.

Below are two examples of reports in Mixpanel you can create after sending an “Annotated Signup” event. Reports like these let you determine not only how many users you are buying, but how valuable the users you bought are actually are.

This funnel uses “Annotated Signup” to determine what percentage of users from different acquisition sources eventually complete level 5 of this game, a key metric of engagement for this publisher.

This report looks at what percentage of users acquired via TapJoy return and continue to use the app 7 days later. You apply different filters for each of your acquisition sources to see which ones are retaining the best.

Once you’ve put the basic infrastructure in place, you can start being more granular. Different campaigns with different messages, creative and placements can be assigned different values for your acquisition source property, so you can optimize your spend to where it is most effective.

What happened

Before explaining exactly what happened, it’s useful to have a rough overview of what happens to an event when it’s sent to Mixpanel. From the client perspective, each event is simply an HTTP request. On Mixpanel’s side, however, each HTTP request hits a few different servers in succession:

1. First, we have HTTP load balancers running on api.mixpanel.com with IP level failover. Each balancer maintains a list of API servers that are currently running and will pick one for each event it processes.
2. Our API servers do basic event validation and optionally add some useful information to the event, such as time if it’s missing or the client IP address if requested. Similar to our balancers, each API server maintains a list of queue servers that are running and will pick one for each event.
3. For queueing, we use piece of software called kestrel, which is a hybrid in-memory / on-disk queue. This is the final step during the initial track request. Normally, these queues hover around 0, meaning that you’ll see your data in real time.

Yesterday, we saw failures on each of these pieces of infrastructure. We spent most of the downtime tracking down the root problem (and we made a few mistakes changing settings on otherwise working servers). Ultimately, we found one queue server with an apparently running version of kestrel, but with the following in the log:

#
# A fatal error has been detected by the Java Runtime Environment:
#
# Internal Error (safepoint.cpp:308), pid=13379, tid=140221886957312
# guarantee(PageArmed == 0) failed: invariant
#
# JRE version: 6.0_26-b03
# Java VM: Java HotSpot(TM) 64-Bit Server VM (20.1-b02 mixed mode linux-amd64 compressed oops)
# An error report file with more information is saved as:
# /tmp/hs_err_pid13379.log
#
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
#

Unfortunately, because the process was still running according to Linux, none of our automated failover systems worked. They rely on TCP/IP level errors, such as would happen if the machine was down, or kestrel wasn’t running. Instead of marking the machine as down, our API servers simply filled up with connections trying to write events to a process that presumably was spending time doing the Java equivalent of a core dump.

Once, we figured this all out fixing the problem was simply a matter of restarting the affected server.

What this means for you

Regrettably, track requests sent during this time were lost. In an hourly report on April 16th, you will see a drop-off in events for the hour of 3pm PDT, and slightly lower events for 2pm and 4pm. Consequently, your daily data for April 16th will be slightly depressed as well.

How we’re going to fix the problem

We are extremely proud of our uptime and stability, and are working to prevent this from happening again. We realize that all sorts of failures happen in the real world and we need to be able to recover from them quickly and transparently regardless of whether it’s a networking problem or a JVM crash. It’s clear that our current detection system is not good enough. The queue server should have been marked down almost immediately.

With that in mind, we’re rewriting our kestrel client code to put strict timeouts on every operation when enqueueing events. Even if we never get any TCP/IP errors, we should still fail and fail fast, so that we can mark the server as down and our API continues to work.

Additionally, it’s clear that our testing does not currently cover all failure scenarios. We’re working on expanding that test coverage so that when we can be sure that the failover systems we have in place will work correctly in production.

If you have any questions about the downtime, please email support@mixpanel.com

Introduction -

Our philosophy here at Mixpanel is that your analytics should embody a scientific approach to solving your business problems. This all starts with planning out your data collection strategy. A little work upfront before your implementation can pay off tenfold when down the line you can instantly access the metrics you need.

The data that goes into mixpanel is broken out into events and properties. Events represent meaningful interactions a user has with your website or mobile app, properties log any type of data that you might later want to associate with that event. Properties can be associated with a single event, or, in the case of super properties, with all events sent by a user.

Which Events to Track –

Seperate signal from noise. Only track events that are important to your business. We’ve found that there is a temptation to make sure that every mouse click or tap should be recorded, especially when folks are just starting with analytics. What we recommend though, is that you just pick five or six meaningful events to start out with.

Begin by thinking about your business model, and the mission of your project. Which actions do users take on your site which affect the bottom line, which move your mission forward? Start with your pivotal events. These are generally easy to come up with. They are generally the same events that your board of directors is interested in! Hint: if you are a profit driven project, these are the events that effect the bottom line. Some examples of pivotal events for different kinds of web and mobile applications include:

• Account Created

• Order Completed

• App Downloaded

• Link Shared

• Photo Uploaded

• Email Opened

Once you have a list of pivotal events, imagine a few “funnels” for these events. Funnels are paths that users might take before completing the pivotal event. Here’s an example.

(1)Viewed item detail→(2)added item to cart → (3)viewed cart
→ (4)entered billing info → (5)order completed

This list of steps is a good starting point for the events that you should track.

Think about other reports that you would like to see. Which events would be required? Allow your reporting needs to drive your event tracking. If you are unfamiliar with mixpanel’s reporting. Attend one of our Webinars, or reach out to us at support@mixpanel.com.

Naming Events vs Properties -

Notice in the above example

(1)Viewed item detail→(2)added item to cart → (3)viewed cart
→ (4)entered billing info → (5)order completed

we have chosen names for these events which are pretty general. On the contrary, we could have listed:

(1)Viewed Product #248677 detail→(2)added item to cart
→(3)entered coupon code → (4)viewed non-empty cart
→ (5)entered US billing info → (6)order completed (coupon)

Obviously, this is a little over the top, but our point is that in the second set of events we’re storing extra data in the event names. Doing this will cause problems when you try to scale, let alone hamper your agility when reporting. Keep your mixpanel event names generalized. Store details in properties.

For example, if your users can download a file from your site, even if you have only one type of file users can download, call the event “initiated download” rather than “downloaded lolcat.gif.” It’s better to store the details of what’s being downloaded in a property for that event. Then when your company expands and adds “potato_like_elvis.gif,” you can easily keep tracking (and reporting to assess the success of your new content) without a hiccup! Keeping meta-data out of your events and in your properties where it belongs will keep your mixpanel implementation scalable and maximize the value you get out of reporting.

To Super or not to Super -

Many properties are static for a given user. Examples are:

• user-name

• gender (at least for the duration of the user’s session on your site)

• whether or not they’ve viewed the FAQ

• which ab testing group they are part of

These are all prime candidates for being registered as super properties. This will save you time on your implementation because even a naked mixpanel.track(“this event”) will be tagged with any super properties already registered for a user.

Any piece of user specific meta-data that is dynamic should be included within the mixpanel.track call. Examples are:

• used coupon

• number of items in cart

• total photos in upload cue

• recipient of message

these dynamic elements are often event specific, like the filename of the file that was downloaded, the user-name of the person messaged, etc.

Getting Distinct-

Issues often arise, especially when defining funnels in which not all the events are sent from the same place, due to a property called distinct_id. Distinct_id is Mixpanel’s way of telling which user to attribute an event to.

By default, Mixpanel will assign users distinct_id’s on the fly, and you could choose to use ours. However, if you want to send an event server-side, or when an email is opened, or across you web app and your android app, you’ll need to find a way to pass the value of distinct_id across those devices. Usually it is easier to register your own distinct_id through the identify function. Use something unique but easy for your app to find, like a user’s email or User ID. Being explicit about distinct_id will help keep your data attributed to the same user across devices.

Good Luck -

Mixpanel is an incredible tool, and it’s getting better all the time. We’re really excited that you’re adding our tracking code to your site and the solutions team is here to help you get the most out of your data. Still feeling confused, drop us a line at support@mixpanel.com, or drop by the site during pacific business hours and catch us on IM.

Previously, you could only do a single segment, asking questions like “Show me my funnel broken down by country.” Now, you can ask much more advanced questions, filtering and segmenting on multiple properties – for example, “Show me my funnel where users are male, greater than the age 40, and break it down by where the user came from.”

Some valuable use cases that came directly from customers include:

• Segmenting your funnel but constraining your segmentation to a particular A/B test. (e.g. Break this down by country but only where users saw experiment “A”)
• The ability to filter on multiple utm_* parameters and then being able to segment.
• The ability to handle specific data types like numbers and lists of things to give you more control over how you “query” your data.

To try it out, all you have to do is flip the toggle beneath the funnel visualization to “Advanced”:

Our goal is to empower our customers to answer some of the hardest questions in the world about their data in the simplest way possible. We hope we created more value for all of you aiming to understand more about your own customers. This most recent addition truly makes funnel analysis with Mixpanel the most powerful and advanced in the industry.

Just so you’re not concerned, here’s some proof (lines 101-157). Instead of using UDIDs, we use a one-way hash of the user’s MAC address which we believe is a reasonable solution for tracking the same user consistently yet allows us to be polite about their privacy. As usual, you can always override a property called “distinct_id” with your own unique identifier (like a user id in your database).

Developers: See the diff that was made 6 months ago.

You find the official documentation for the API here: https://mixpanel.com/docs/api-documentation/exporting-raw-data-you-inserted-into-mixpanel

Here are some useful examples of ways you can use this to your advantage:

• If you’re advertising on mobile, you can export and analyze all the UDIDs that advertisers are claiming they got for you. This helps you with attribution and de-duplication if you’re working with multiple advertisers.
• If you saw a spike of 10,000 events but noticed only a few unique users contributed to it, you can now dive a bit deeper to determine what exactly happened.
• If there’s a particular feature Mixpanel can’t do (we’d love to hear about what that case is by the way: support@mixpanel.com), you can always feel safe knowing there’s a way to get it. You can avoid having to store the data on your side manually from now on.

Feature highlight

One feature we would like to highlight is the ability to filter on a property or segment. Not only can you say “I want all the data from this date to this date” but you can also say “I want all this data from this date to this date where my property source = google.” Check out the “where” parameter in the documentation which can handle all kinds of expressions.

As always, please let us know if you run into any problems by emailing support@mixpanel.com.

Testing is an essential part of life. We touch the water with our toes before we jump in. We taste a piece of new food before we eat the entire meal. We test drive cars and skis before we buy them. In general, we try to collect information when it is available to make future decisions about larger investments. In some ways, measuring cause and effect is easy: When you can hold “everything else constant”, its easy to determine that a new pair of sneakers gave you blisters and a shampoo you used gave you dry scalp and not the other way around. But in other cases it is more difficult: was it the coffee or the wine or the soda you drank yesterday (or a combination of all three) that gave you a stomach ache last night?

When building software that is used by many customers the same problem presents itself. Was it the new algorithm or the new module that generated an increase in average engagement for users? Did the new user tutorial or the coupon offer drive a higher conversion rate? One way to isolate the effects from these potential sources is to introduce them one at a time and see what measures change. But this is not realistic: other businesses who don’t test this way will move much faster and a subset of these will beat their competition. Also, pushing new features out one at a time does not account for hourly, weekly or seasonal effects.

What do you do then when you need visibility on the impact of product features and insight on where to take the next steps? The answer: experiments. Experiments are actually ideally suited for rapid development in the web and software industries that are filled with a large number of fluid users. Why?

1) It is easy to segment users randomly based on a numerical identifier (by taking the modulus of that identifier for example).

2) Software and particular features are easy to execute on those segments (if id meets a condition then show new feature).

3) It is easy to measure results and compare segments if you are logging raw data of user behavior.

Employing the scientific method ensures the credibility of the experiment. This means forming a hypothesis, implementing a control group and collecting the right data to make the measurements you need to draw conclusions. However, in competitive markets, there is not much room for retesting and verifying results as the scientific method encourages. In other words, a compromise between experimental verification and intuition is a far more successful strategy for business than intuition alone.

We are very proud of the accuracy of our data, and are extremely sorry that this error occurred. We know that you make key business decisions based on the data you see in Mixpanel, and even though the miscounting only lasted for twenty minutes, we wanted to make you immediately aware of it. The rest of this blog post will discuss the full details of this error and possible ramifications for your decision-making.

The over-counting impacts all reports on Mixpanel.com or any API calls that request a total count for any event and include 6pm PST on 1/16/12 within the time period of the query. However, this over counting does not impact queries for unique event counts. This means that the Funnels report and the Retention report, which are entirely based on uniques, are completely unaffected. In addition, any queries in the Segmenation or Trends report in uniques mode are not impacted either.

How can I adjust my data to account for this?
In most cases, we do not recommend trying to adjust your data to account for this error. For daily reports, the difference will be trivial. For the day of January 16th total event counts will be roughly 5% higher than they were in actuality. For monthly reports, the total event counts will only be 0.1% higher than they were in actuality. The only case where you might want to adjust the data coming out of Mixpanel is in the case of hourly reports. If you are basing a decision on an hourly report, then divide the count for your total events that happened during the hour of 6 PM by 2. The result will be very close to the true event count for that hour.

Will I be billed for these data points?
Absolutely not. You will not be charged for any overages to your plan caused by this error.

I’m a geek – tell me what really happened
First, it’s necessary to describe a small part of our infrastructure. When a user sends an event to Mixpanel, we do a small amount of validation — mostly checking for syntactical correctness — and then immediately put the event on a queue. Under normal circumstances, the number of items on the queue stays very close to zero, meaning that within seconds of sending an event it should show up in your reports. However, decoupling receiving events from processing them allows us to easily perform server maintenance that would otherwise require significant downtime.

For a long time now, we’ve had multiple queue servers so we aren’t reliant on a single machine, but we haven’t had automated failover. In practical terms, that means that if a queue server goes down in the middle of the day we can do a manual failover within minutes, but if it goes down in the middle of the night, it could be quite a bit longer before we can switch everything over. The change we pushed out Monday was intended to remedy this situation. Basically, when an event comes in, we try each queue server that we currently think is up one at a time until we successfully enqueue an item. Unfortunately, our code to check whether putting an item on a queue was successful or not was incorrect and consequently each event was added to each queue server (currently, there are four). We noticed the problem almost immediately and had the fix within 20 minutes.

What we are doing to keep this from happening again
Unfortunately, although we have queueing related tests, in our test environment there is only one queue server and so none of our tests caught this particular problem. That particular hole will be fixed in the coming days.

Once again, our most sincere apologies and regrets for this error. If you have any questions please do not hesitate to reach out to us at support@mixpanel.com.

Today we are rolling out our new retention tracking system. Now, instead of choosing a user’s cohort on an event-by-event basis (the first time the event was fired) you tell Mixpanel their cohort by sending a $signup event (in fact, this event can be called anything, the key is that you only send it once per distinct_id. If you call it $signup there will be a slight improvement in performance for your report). Usually you would send this event when they first make an account, but a lot of our customers have been asking us about their existing users. How can you tell Mixpanel which cohort to put your current users in?

You can do this, but there are a couple prerequisites:

• You must be already be using mpq.identify() or distinct_id to keep track of your users within Mixpanel
• You must have an existing record tying the identify() or distict_id value you have been sending to Mixpanel with a date you want to use as a user’s birthday. For example: a users table in your database with user_id and date_created fields.

If you meet those prerequisites, you can write a one-off script to send the $signup event to Mixpanel for all your users (each request will count as a Mixpanel data point for billing purposes). To do this, for every user in your database you would make a GET request that looks like this:

This request is very similar to our standard HTTP API (documented here). The data parameter is a Base64 encoded JSON array with the event you are importing ($signup) and the associated properties. By decoding the Base64 data parameter from the above request you can see this:

{u'event': u'$signup',
 u'properties': {u'distinct_id': 481,
                 u'time': 1321499371,
                 u'token': u'13fe3ddc86eb6f90c4ee7d0d47563150'}}

• The event is $signup
• The ‘distinct_id‘ property is the user ID you have been sending to Mixpanel up to this point for that user. For Javascript Library users – this is the value you put in mpq.identify()
• The time property determines which cohort the user will be in. It is in Unix Epoch format (seconds since 1970). Times should be GMT. The above example, 1321499371, represents November 17th at 3:09 AM GMT.
• The token property is your Mixpanel project token.

There are three differences between this import method and the regular HTTP API:

• this method lets you import events older than 31 days
• the endpoint is /import/ instead of /track/
• as an added level of security, you must include your API key as a parameter outside the Base64

The /import endpoint works for any events you would like to import – not just the $signup event.

You can download a sample PHP script that you can modify with your own data, token and API key to send $signup events to Mixpanel.

If you have trouble importing $signup events for your current users, please reach out to support@mixpanel.com.

To our existing customers, the biggest change is in the way cohort groups are determined. In the past, you could only select a single event for retention analysis, now you can select two – one for determining the cohort (usually you would pick a signup or $born event for this) and one to look at the retention for. This is explained in detail on our documentation page, and in the video below the fold.

You may be wondering: “what happened to compounded retention?” Now all retention is compounded retention – when you pick the first event, we will include all users in the cohort who fired that event, not just those who fired it the first time. To see birth retention, you need to fire an event that will only be sent once and determine the birthday for each user. This event can be called whatever you want, but we recommend calling it $born or $signup. To learn how to properly send a $born or $signup event, read this blog post. To learn how to import these events for your existing users, we have a blog post about that as well!

If you have any questions about new retention, we will be hosting a Live Q&A Webinar tomorrow at 11 AM PST. To tune in visit http://www.mixpanel.com/webinar then.