What to do if you drank the Kool-Aid on bullshit metrics

I've always been interested in quantitative data and the ability to derive insights from that data. So when a VC Firm (Andreessen Horowitz) raised a ton of money ($10.25M) for an analytics start-up (Mixpanel), I took notice. Soon after raising the money, Marc Andreessen and Suhail Doshi came out swinging with a punchy line aimed at getting to the heart of a common problem in the technology industry:

Some people call page views and the like “vanity metrics,” but Marc Andreessen and Mixpanel founder Suhail Doshi have decided they want to raise the shame level by calling them “bullshit metrics.”

Andreessen told me in an interview last week, “People think they’re richer if they have Zimbabwean dollars than U.S. dollars.”

“We and other investors need to get more vocal,” Andreessen said. “Page views and uniques are a waste of time.” 

Andreessen said his firm won’t throw start-ups out the door if their pitches include bullshit metrics - but it’s perhaps something they might consider.

- Liz Gannes @ AllThingsD

So if you drank the Kool-Aid and decided that random download stats or pageview metrics, that go up and to the right, are pretty much worthless, then where do you turn? What do you measure that is more insightful than these bullshit metrics?

I've written previously about Dave McLure's "Startup Metrics for Pirates: AARRR" and its definitely worth starting there for an overarching framework for how to think about the whole customer lifecycle. Once you understand that lifecycle for your particular product, you can then begin to integrate an analytics platform into your product that captures the information you need and can then act on. Event based analytics (the kind of thing that Mixpanel excels at) is based around capturing and then segmenting all the events that your users perform. By segmenting the aggregate of these events you are then able to build an awareness of and insight about who your customers are, how they first got introduced to your product and how they are currently using it. Segmentation is a great starting point but there is another even more valuable tool called cohort analysis once you have all your events setup. Cohort analysis allows you to measure customer retention so that you can answer the question of whether or not your customers love your product. Andrew Chen has a great blog post on this where he asks that very question.

So once you have a few months worth of cohort data, how can you then determine whether your product is above or below par? It turns out this is a very hard question to answer because it usually depends on the type of product, your customers and a variety of other factors (essentially there is no "standard" that's works for every product). This doesn't stop some people/companies from speculating so here are a few reference points:

• Suhail Doshi of Mixpanel answers the following question on Quora in terms of 1 week retention: What are typical good retention rates for consumer websites?
• Fred Wilson, VC and principal of Union Square Ventures, has a 30/10/10 rule where 30% of users are retained. 
• Flurry, analytics platform for apps and games, wrote a post titled "App Engagement: The Matrix Reloaded" which contained the following matrix for various product categories and has a variety of retention rates over 3 months:

So as a very general rule of thumb, a retention rate of 30% month after month seems like a decent number to benchmark against. But a word of caution, definitely don't consider that number to be some special threshold for which your product can be deemed successful in the marketplace if you surpass it. The matrix from Flurry above was created in Oct, 2012 but an earlier version first appeared back Sep, 2009. If you look at how the retention rates for social networking apps have change over the last 3 years its startling. Back in 2009 social networking apps had a 90 day retention rate of approx. 15% as opposed to approx. 34% in 2012. 

As always, in the technology business, the goal posts continue to move every single year. Ben Horowitz, of VC firm Andreessen Horowitz, said this:

The technology business is fundamentally the innovation business. Etymologically, the word technology means “a better way of doing things.” As a result, innovation is the core competency for technology companies. Technology companies are born because they create a better way of doing things. Eventually, someone else will come up with a better way. Therefore, if a technology company ceases to innovate, it will die.

RESAAS reblasts App Featured on Appcelerator Titanium Blog

In a previous post I showcased the iOS and Android App that my team and I at RESAAS released in March, 2013 after only 2 months. The App was built using Appcelerator's Titanium cross-platform framework after we migrated from an older PhoneGap implementation. We chose to go with Titanium due to its (almost) write once run everywhere framework.

Once we released the App, the folks over at Appcelerator took notice of it and loved the look & feel of the App, specifically the photo heavy activity feed that showcases real estate professional's listings. They subsequently asked me to respond to a number of questions they had about our App for an upcoming blog post on their developer blog. 

Here is that blog post they created: http://developer.appcelerator.com/blog/2013/06/featured-app-reblasts.html

The official RESAAS blog also has a couple posts about other features related to reblasts App:

• Taking and sharing photos: http://blog.resaas.com/2013/05/iphone-app-update-take-photos-and-share.html
• Commenting on reblasts: http://blog.resaas.com/2013/05/iphone-app-update-commenting-on-reblasts.html

Mixpanel Implementation of Startup Metrics for Pirates: AARRR

Dave McClure of 500 Startups (a seed accelerator and investment fund) has a great slide presentation on slideshare called Startup Metrics for Pirates: AARRR!!!. Don't let the old school graphics fool you, it's packed with a ton of insight about how to strategically think about your startup in terms of quantifiable metrics.

Startup Metrics for Pirates from Dave McClure

On a previous post about Chamath Palhapitiya and focusing on the right things I included his simple 4 stage growth framework which had the following stages: Acquisition, Activation, Engagement & Virality.

Dave's metrics, called AARRR, have an additional stage (5 in total) and are as follows: Acquisition, Activation, Retention, Referral and Revenue. In Dave's case the Referral stage is similar to Chamath's Virality stage except for the fact that Dave goes into some detail about using it to effectively acquire new users on the back of your existing users. Chamath advocates quite passionately to not even focus on the concept of virality due to its illusive nature and the negative distraction it will cause. There are some very subtle differences between virality and referral but I will discuss those in an upcoming post.

Chamath's team at Facebook never spoke about virality or k-factor while building out their incredibly successful social network. He believed it was essential for his team to focus on the quality of the actual product and continue trying to make the overall experience for users better and better. Chamath says far too many people focus on this holy grail of trying to make their "bad" product viral in some way instead of finding ways to make their product better (or even just decent/okay) so users actually keep using it over time.

Either way Dave's startup metrics for pirates (described in the presentation above) give a solid footing to any startup interested in building a quality product/service and a business around that product/service. Measuring how successful things are going during a startups lifecycle is difficult due to the dynamic nature of a startup, but this simple framework and associated metrics are sufficiently generic enough that they are still relevant as a startup morphs into a revenue generating business. 

I am a huge fan of Mixpanel's event based analytics platform. I use it extensively at RESAAS and love the ease of setup and ability for both engineers and marketers to easily sift through tons of data, explore theories and then develop insights that impact future product decisions. If Dave's startup metrics for pirates, AARRR, is something you decide to build yourself then I highly suggest trying Mixpanel as the analytics platform behind those metrics (other product you could use include Google Analytics, Kissmetrics, Woopra, Flurry... etc). What's nice is that the Mixpanel team even put together this great blog post back in November, 2012 about using Mixpanel to implement Dave's AARRR metrics.

Growth and Distribution

One of the many joys of working in technology company is seeing your product used by actual people. Right from the beginning when the initial beta users start signing-up there is a sense that all the hard work (thinking+coding+coffee) might come to something. Then suddenly as user concentration increases the first few sparks of social interaction between users begins. Finally, when traction sets in, entire communities of people start engaging around an idea and then the next idea and on and on it goes.

I got my first real exposure to rapid scale and the interaction between a swarm of users when I built a scavenger hunt game called SRCH for Starbucks. The game was coded, tested and ready to go when Starbucks released their first clue. Within minutes tens of thousands of people descended on our application trying frantically to win one of the coveted prizes. They frequently turned to Twitter to communicate with each other where they'd trade strategies, vent their frustration for losing or gloat when they won.

Since starting at RESAAS back in 2011 as employee #1, we've all worked really hard to build an enterprise social platform that works well within the dynamics of the real estate industry. From just a handful of beta users in the early days to a thriving community of real estate professionals today, RESAAS has grown by leaps and bounds. 

Great products need great technology but even more than that they need great distribution. Simply finding low cost channels that put your intended audience in front of your product is not enough. The competitive advantage exists for those companies who are able to optimize around the conversion and retention of newly acquired audiences. It requires deep analytical insight about actual user behaviour along with a rigorous process for testing, iterating and optimizing at a rapid pace. This is why we setup a Growth Team at RESAAS. It's purpose was and still is to accelerate user growth across the platform. To illustrate how far we've come from just a handful of beta users, the following real-time data shows RESAAS user activity across the US: