I've always been interested in quantitative data and the ability to derive insights from that data. So when a VC Firm (Andreessen Horowitz) raised a ton of money ($10.25M) for an analytics start-up (Mixpanel), I took notice. Soon after raising the money, Marc Andreessen and Suhail Doshi came out swinging with a punchy line aimed at getting to the heart of a common problem in the technology industry:
Some people call page views and the like “vanity metrics,” but Marc Andreessen and Mixpanel founder Suhail Doshi have decided they want to raise the shame level by calling them “bullshit metrics.”
Andreessen told me in an interview last week, “People think they’re richer if they have Zimbabwean dollars than U.S. dollars.”
“We and other investors need to get more vocal,” Andreessen said. “Page views and uniques are a waste of time.”
Andreessen said his firm won’t throw start-ups out the door if their pitches include bullshit metrics - but it’s perhaps something they might consider.
- Liz Gannes @ AllThingsD
So if you drank the Kool-Aid and decided that random download stats or pageview metrics, that go up and to the right, are pretty much worthless, then where do you turn? What do you measure that is more insightful than these bullshit metrics?
I've written previously about Dave McLure's "Startup Metrics for Pirates: AARRR" and its definitely worth starting there for an overarching framework for how to think about the whole customer lifecycle. Once you understand that lifecycle for your particular product, you can then begin to integrate an analytics platform into your product that captures the information you need and can then act on. Event based analytics (the kind of thing that Mixpanel excels at) is based around capturing and then segmenting all the events that your users perform. By segmenting the aggregate of these events you are then able to build an awareness of and insight about who your customers are, how they first got introduced to your product and how they are currently using it. Segmentation is a great starting point but there is another even more valuable tool called cohort analysis once you have all your events setup. Cohort analysis allows you to measure customer retention so that you can answer the question of whether or not your customers love your product. Andrew Chen has a great blog post on this where he asks that very question.
So once you have a few months worth of cohort data, how can you then determine whether your product is above or below par? It turns out this is a very hard question to answer because it usually depends on the type of product, your customers and a variety of other factors (essentially there is no "standard" that's works for every product). This doesn't stop some people/companies from speculating so here are a few reference points:
- Suhail Doshi of Mixpanel answers the following question on Quora in terms of 1 week retention: What are typical good retention rates for consumer websites?
- Fred Wilson, VC and principal of Union Square Ventures, has a 30/10/10 rule where 30% of users are retained.
- Flurry, analytics platform for apps and games, wrote a post titled "App Engagement: The Matrix Reloaded" which contained the following matrix for various product categories and has a variety of retention rates over 3 months:
So as a very general rule of thumb, a retention rate of 30% month after month seems like a decent number to benchmark against. But a word of caution, definitely don't consider that number to be some special threshold for which your product can be deemed successful in the marketplace if you surpass it. The matrix from Flurry above was created in Oct, 2012 but an earlier version first appeared back Sep, 2009. If you look at how the retention rates for social networking apps have change over the last 3 years its startling. Back in 2009 social networking apps had a 90 day retention rate of approx. 15% as opposed to approx. 34% in 2012.
As always, in the technology business, the goal posts continue to move every single year. Ben Horowitz, of VC firm Andreessen Horowitz, said this:
The technology business is fundamentally the innovation business. Etymologically, the word technology means “a better way of doing things.” As a result, innovation is the core competency for technology companies. Technology companies are born because they create a better way of doing things. Eventually, someone else will come up with a better way. Therefore, if a technology company ceases to innovate, it will die.
No comments:
Post a Comment