A month ago I got a chance to speak at Tech in Asia Product Development Conference in Jakarta about data-driven versus data-informed. Post-conference I was approached a few times to discuss more and speak at other events about the topic. Since people attended my talk found it useful, I thought I might as well share it with the global community.
Recently, the term data-driven started to become a buzzword, adding on to the list of other cool, techie jargon: disruptive, pivoting, design thinking, and many more. So, let me first attempt to get us all on the same page.
Being data-driven means letting data be the center of a team and company’s decision making process. Data then plays an essential part of your company. Decision makers reply mostly (and sometimes solely) on data. Some decisions can be made without human being involved.
Some of the conversations that came up from a data-driven company are:
- “Let’s A/B test this, and implement the version with the better result”
- “Don’t talk to me unless you have have data”
- “We’ll just follow what the data tells us,”
The truly data-driven organization will implement this process across teams and functions. This means that every team (product, marketing, customer success, operations, etc.) uses data intensively for all their decisions.
So, what are the benefits?
It eliminate a lot of human biases involved in decision making
It is a lot less time-consuming because we take human out of the equation
How many of you have been in a 3-hour long meeting that people argue back and forth with no sign of compromise? Even better if you work in at a startup. Sometimes we discuss and argue for the sake of doing it, then the HiPPO made the final decision. (Yep, I know how you feel).
In fact, a study from MIT has proven the benefits of being data-driven. Prof. Erik Brynjolfsson and his colleagues studied 179 large publicly-traded companies and concluded that those companies are 5% more productive and profitable than their competitors.
However, being data-driven also has its drawbacks.
It requires a huge amount of data for the decisions to be accurate
You have to watch out for outlier distortion. Outliers are a data point from which are radically different from your average data. In gaming, positive outliers are considered “whales.” Those customers whom make either big purchases or make them very frequently, generating a significant amount of revenue for the company.
At my previous company, we even coined the term, “whale hunting.” In other industries, it might be those customers who use your product to solve a different problem than most of your customers. Their behaviors are so different, thus should be excluded from your data to make decisions about your average customers. Few data points together with the outliers can lead your company toward the direction that you don’t want to go.
It requires someone with data science knowledge and a lot of resources to be beneficial
For startups and big organizations that just started Data Science department, it might be hard to be completely data-driven due to lack of capabilities and resources. Current employees might not have enough knowledge to build the infrastructure to support. Some might not even have anyone who has expertise with data. At my previous company, we have 10 data scientist and at Kulina, we have 0.5 person (myself and our head of tech).
There is still a bias in the way we gather data
Most of times, what people say is not what they do. So, if we gather data by asking them implicitly, we risked making decisions on the wrong information. For example, during the scandalous time of Uber and the #deleteuber movement, most of my friends said they would never use Uber again. When Lyft (its main competitor in the US) has a surcharge, they went back and ordered Uber for their regular rides!
Let’s look at an example from my experience
Another great example is when we were trying to figure out which social media platform we should spend more time and money to engage with our users. We sent out a survey asking them, and the result was quite surprising. We saw Facebook on top of the list, which is expected. However, what puzzled us was the fact that Google+ was second on the list (no hard feeling to the team at Google).
A few team members asked if I would agree with letting them spend more time building our Google+ presence. “Maybe we didn’t know our users well,” a team member mentioned. Being data-driven, I would probably greenlight this. But being stubborn as usual, I discussed with another product marketing manager on what could’ve gone wrong.
We came to a conclusion that our users might not understand the difference between Google and Google+. In fact, when we look deeper into the way the question was asked. Instead of using the icon, we used the wording Google+ in the questionnaire. If you didn’t pay close attention to the survey, you probably thought we were asking if you use Google!
So, what is a data-informed decision?
Making data-informed decision takes data only as a factor that can be dismissed from time to time. This kind of decision making allows other factors such as customer experience, gut feeling, brand consistency and the HiPPO to take the lead.
The benefit is that data is put to be challenged
Because we don’t purely base our data, we use it to create a hypothesis. No matter how good data is, it has its limitation. It is just a snapshot of reality that doesn’t paint the full picture of our customer journey and behavior. We are forced to user other factors to help make better decisions, including our own judgement.
One of the example was at my previous company. To give you a little bit of background information, the nature of our gaming business was having great revenue on the weekend because that’s when our users played games. On a Monday, one of the new product directors approached me regarding a low weekend revenue, and he was very worried.
With some experience and guess, I suspected that our revenue was down because we ran sale the prior weekend. The same effect occurred for the past five times we ran sales. If we look at the same cycle before, our average was actually better. In fact, we figured out later that it was because one of the marketing campaigns that were launched to encounter this specific problem.
Facebook’s Newsfeed version that moved the key metrics was actually done without being data-driven
Adam Mosseri and his team back then decided to create the newsfeed without data. In fact, it got a lot of negative reactions that users were angry and even created a Facebook group I AUTOMATICALLY HATE THE NEW FACEBOOK HOMEPAGE (watch his talk here).
“At the end of the day, we have to have a gut to make bold decisions without data.” Adam Mosseri
This doesn’t mean that data-informed decisions don’t have their drawbacks
It requires a lot of time to discuss and analyze, and conclusions could not be formed easily. But remember at the end of the day:
“All the data in the world won’t fix a fundamentally bad product.” Andy Carvell.
In order to be innovative and build the right product for our customers, we cannot just sit in the room looking at rows of data and charts. Hint: Data doesn’t have to always be big data. Gather qualitative feedback by talking to your customers, observing customers struggles through usability tests and figuring out the jobs customers are hiring you for are key to building products.
So, my recommendations to you is
For smaller companies, you should always aim to be more data-driven. You probably don’t utilize the data you collect (or even collect them at all). The more data you have, the better hypotheses you will generate.
For larger companies, you should be able to make bold decisions that not move the key metrics but also improve customer experience. We should use data to help us make better decisions, but don’t reply on it a hundred percent!