How to Pick Metrics that Matter for Your Startup

When founders, investors and technology experts evaluate early-stage startups, one of the key things they focus on is a set of commonly understood metrics. They ask questions like these: How quickly is the user base growing?  Will that growth continue? How much revenue is the company bringing in? Is revenue growing, and if so, how quickly?  

Standard metrics like revenue and user growth are important because they provide critical information about a company’s health and future prospects. For public companies, they’re critical.  But let’s make something clear: when you’re talking about startups, metrics that tickle the limbic system - like money (revenue) and proxies for popularity (growth) - do not necessarily prove that a company’s mission is more likely to be achieved in the long run.

It’s quite easy to rattle off growth and revenue statistics.  It’s much harder to (1) explain a startup’s investment thesis carefully and responsibly, (2) present the evidence corroborating that thesis to date (and what evidence, if any, undermines it), and (3) evaluate what needs to be done to achieve the startup’s longer-term vision.

Understanding what really needs to go right is hard work - but it’s worth it.  The key thing to focus on is a startup’s investment thesis.





 

What is an Investment Thesis?

An investment thesis does two things:

First, it crystallizes the founders’ understanding of the company’s core aims.

Second, an investment thesis articulates the general approach the company will take to fulfill those objectives.

For example, Palantir’s investment thesis might be something like this: 

We endeavor to create a data fusion platform by ensuring that our software (a) can handle data of any type and from any source and (b) allows human beings to easily interact with multiple data streams to solve difficult problems.


An investment thesis spells out in general terms how the company is going to build its product or service. In Palantir’s case, the company’s long term objective is to create a data fusion platform capable of handling information from any source.  This implies that as the company moves forward, it’s got to get better at fulfilling this goal.  And the same can be said about enabling cooperation between its software and the human agents that operate it.  
Over time, an investment thesis serves as a guide to ensure that the company is on the right track.  And this is why building an innovative company is so challenging: you have to focus on your mission myopically, even obsessively. If moving metrics is your sole mission, it is highly unlikely you’ll create anything new. It is also probable that your company will ultimately collapse, rather than become and remain a treasure.

 

How do Real Companies Corroborate their Theses?

The best companies do not try to optimize the metrics that Silicon Valley is so fond of focusing on.  They figure out what they’re really trying to accomplish and then determine whether their efforts are actually successful by carefully comparing results with their predefined goals.  The delta between these original goals and the actual results provides critical feedback for founders who are obsessive about building innovative products and services.  To see how this works in action, we are going to consider Tesla (electric vehicles), Soylent (next-generation food) and Omada Health (digital therapeutics).

 

Tesla

Tesla is on a tear.  But it’s easy to forget that when the company was getting off the ground, many pundits in technology and transportation didn’t think Elon Musk had a fighting chance.  So we are going to ask the following question:  When Musk decided to build Tesla, what was his investment thesis?  Did he originally intend to build an electric car company that would sell 40 to 50K Model S sedans per year worldwide? No. The original investment thesis was to create an electric car that would met certain safety and range requirements.

Stated formally, Musk’s thesis might have been something like this:  

Tesla endeavors to create an electric sports car that (a) meets all safety and manufacturing requirements imposed by relevant regulators and (b) provides a range of travel suitable to consumer expectations.  

Having distilled Tesla’s early objectives, we can now ask why there are no aspirational growth metrics.  The answer is simple, but only after you hear it:  You could sell Tesla Roadsters to hundreds of thousands of customers and raise enough venture capital to open Tesla dealerships across the entire globe and still fail to create a product that met the safety and range requirements set out in the investment thesis.  Those requirements reveal what Tesla absolutely needed to do in order to succeed; the metrics - including customer growth and dealership footprint - are byproducts of nailing these two requirements.

What was true of Tesla in its early days is also true of two much younger startups, Soylent and Omada Health.  Soylent’s mission is to create food that’s extremely convenient, inexpensive, and healthy.  Omada Health is trying to help ordinary people who are at-risk for chronic disease take practical, effective steps to reduce the likelihood of getting sick.  These companies have different missions, but they share a similar challenge:  If they focus on metrics too early - including user growth, revenue, retention, and the like - they will fail to realize the goals embedded in their original investment theses.

 

Soylent

Soylent has arguably already shown that it can provide a food replacement that’s convenient and inexpensive.  However, to corroborate its investment thesis, it has to show that its product has the desired impact on human diet and health.  This is a very tall order.  Soylent’s food replacement consists mostly of soy protein, algal oil, isomaltose, and a variety of vitamins and minerals necessary to healthy physiological functioning.  The question, however, is whether the human body can efficiently absorb the nutrients contained in Soylent, especially at different levels of consumption relative to other foods.  

For example, it could be the case that Soylent absorbs optimally when it replaces a specific portion of a person’s diet.  An individual’s ability to efficiently absorb Soylent, incidentally, could fluctuate as a function of overall health, genetic profile, and levels of physical activity. The point that deserves emphasis here is that every financial metric that people want to cling to when describing Soylent’s future as a company - revenue, growth, retention - is a function of whether the original investment thesis is being achieved by the company.  When the question is whether Soylent will succeed in the long run, there are no shortcuts.  Each element of the company’s investment thesis has to be validated.

 

Omada Health

The same is true of Omada Health (“Omada”), a company committed to helping people at risk for chronic disease take steps to improve their health.  Omada has a 16-week program called Prevent (delivered via mobile devices) that matches users with coaches who assist with meeting health goals.  Some of these goals relate to diet and some center around changing one’s approach to physical health.

Omada believes that changing one’s diet and overall health is significantly easier when empathy is part of the equation.  That’s precisely why coaches and groups play such a central role within Prevent:  Coaching and genuine emotional support from a group of caring peers are hallmarks of Omada’s approach to preventative health.  It is not going to be easy to establish that emotional empathy - of the kind and degree necessary for lasting behavioral change - can be delivered via mobile.  But trying to carefully test this idea is exactly what Omada Health is trying to do.  Leveraging real empathy to produce improved health for users is a central part of the company’s investment thesis.  Just as Tesla in the early days focused on vehicle safety and range and Soylent is now focusing on nutritive value, Omada is trying to prove that empathy actually has a measurable impact on health outcomes.

 

So Which Metrics Matter?

There is a unifying theme present in each of the examples we have examined:  in each case, the founders made meaningful progress by focusing on a set of difficult goals and incrementally proving that those goals could be achieved.

These goals cannot be neatly classified as revenue or growth or retention.  And that’s precisely the point: the metrics that matter are the difficult goals your team is trying to realize.  You might go further and say that the metrics that matter aren’t really metrics at all, they’re testable elements of your overall hypothesis. Reflect, once again, on the bold thesis that a venture-funded startup could build a viable electric car that could potentially compete with powerful, well-capitalized domestic and foreign car manufacturers.  Qutie the thesis, wouldn’t you say?  And yet Tesla proposed and then built a fully-electric vehicle that met all safety requirements and key consumer expectations around travel range.

How can the broad goals put forth by Soylent and Omada Health be measured?

In Soylent’s case, the company’s investment thesis has three prongs. The product must be convenient, inexpensive, and nutritious.  The third prong - nutritiousness - has not been fully established.  So the question remains, what metrics help corroborate that prong?  If Soylent is going to provide comprehensive nutrition for people, then it must meet three requirements (or metrics):  (a) maintain key vitamin and mineral levels in the human body without additional supplement by traditional food, (b) reduce overall levels of dietary toxins (sugars, saturated fats, cholesterol) to healthy levels, and (c) not trigger or increase the probability of other diseases or conditions developing in the human body.  In other words, Soylent must add something (vitamins and minerals), subtract something (dietary toxins) and avoid causing something (diseases not caused by standard food).   

In Omada Health’s case, the company’s investment thesis is that individuals can combat chronic disease more effectively when they receive support from coaches and peers.  The specific claim made by the product team is that the empathy provided by coaches and peers is the driving factor of desirable health outcomes.  Empathy, like many psychological phenomena, isn’t directly measurable.  But you can infer the presence of it by employing the right statistical tools in a properly designed research study.  Omada is unlikely to take this route for ethical reasons, since a properly designed study would withhold coaching and positive peer reinforcement from a control group.  

A more likely strategy would be to carefully collect data for early customers and leverage that data to show that Omada’s Prevent program delivers results - even if those results cannot be scientifically traced to the presence or quantum of empathy provided to participants.  As it happens, this is part of Omada’s approach to marketing today.  The company touts an 80% completion rate for its Prevent product.  Now, scientifically this does not prove much.  It just says that 8 out of 10 people log into the program for the full 16 weeks.  Establishing that the Prevent product is having a causal impact on health outcomes requires a much closer look at users’ health outcomes.  Specifically, two metrics should be carefully tracked here:  (a) the statistical differences between (i) health outcomes of Prevent users and (ii) health outcomes of non-Prevent users (from similar demographic groups), and (b) the statistical relationship between (iii) the amount and frequency of support (empathy) given to users and (iv) those users’ health outcomes.  In a nutshell, Omada must show that its Prevent product is able to produce better health outcomes than other health and wellness programs for similar demographic groups.

 

Does this Sound like Science?

Well, in many ways it is.  The key similarity between validating an investment thesis and testing a theory is that both require rigorous testing of the unique claims being proposed.  This is why Soylent must demonstrate that its product is nutritious and Omada Health has to prove that empathy drives improved health outcomes.  It’s all about corroboration.  And that’s really the key message here:  Corroborating a startup’s vision isn’t for the faint of heart; it’s for people with persistence and a stomach for trying over and over again to do hard things.