Data, decisions, egos and pinball.


Time to read:

6 minutes

The experimental pinball machine

“Each new data he encountered in product impelled him in a direction that fully convinced him of its rightness, but then the next new (anecdotal) evidence loomed up and impelled him in the opposite direction, which also felt right. There was no controlling narrative: the product seemed to himself a purely reactive pinball in a game whose only object was to experiment for the sake of experimenting.

I hope Jonathan Frazen does not mind that his text has been heavily altered. That still seems to be the case with many product teams: We want to be data-driven, data-informed or data-led, but we can not seem to get anywhere because we do not know how to define and use metrics. We struggle to understand the (political) value of anecdotal evidence without understanding the context. We muddle through the murky sea of experiments that give us lots of confirmation and convincing evidence, but little real understanding. What we really want are concrete, actionable insights.

The purpose of data and the two mistakes

Let us start with a simple use case: In the product context, we use data to improve the quality of our decisions.

The first mistake we make is to assume that a sufficient amount of data makes the decisions self-evident, so that we do not have to take risks and the desired results are guaranteed. But even the most accurate measurements are just that: approximations that attempt to accurately represent some aspect of reality. “All models are wrong, but some are useful”, to quote a statistical aphorism. Therefore realise from the outset that some decisions are bets based on probabilities and assess early on how much confidence you have. Acknowledge the uncertainty and incorporate it into your decision-making process.

The second mistake we make is that we judge the quality of our decisions by their success rates. On the surface, this seems plausible (good outcome = good decision). However, there is a clear difference between speculation and correlation. A good decision is the result of a good process that adequately reflects the facts, context and evidence available to you, not just a lucky guess. Whether you can make better decisions and thus increase your chances of better results in the long run depends on your ability to distinguish between these two aspects.

Two scenarios when data is self-explanatory

Data only offers you an obvious next step in two scenarios:

1. You have a constant flow of data and work on small, refined user experience goals. You have a solid understanding of context and user needs. You do not make decisions per se, but analyse data and synthesise it into a user experience.

2. Instead of using data to learn, you need it to support your point of view. We have all been there. Even if there is a mountain of external data, research, etc. to back up the point of view, there are instances where you need to conduct your own experiment to make a point in an otherwise opinion-based decision-making process.

This brings us to the two most important points: the possibilities and limitations of data:

Data does not provide meaning or purpose

It only serves to verify that your assumptions are correct. Therefore, it is not a good idea to use it to make transformative decisions that would often benefit more from a strategic product direction based on more comprehensive research.

Data alone will not change your product

To ensure that experiments are meaningful, think about how the outcome will impact the product before running them. Without this foresight, you risk bouncing around aimlessly like a pinball in a machine.

The factors beyond our control

However, there are factors beyond our control that can hinder this process. Our perception of the world is the result of the way we see and try to understand it, which is based on our values, fears, intuitions, prejudices and pride. What all factors have in common is that they always lead to the same outcome – one that is consistent with your beliefs. Recognising and considering these factors requires humility and a willingness to challenge your assumptions, which can be a difficult but essential aspect of data-driven decision making.

The factors:

Conviction

What evidence would you need to change your mind? This is one of the most important questions you should ask yourself. When you are working on something and investing time and effort, you probably have many beliefs, assumptions and ideas about the problem you are trying to solve, causing you to ignore data that contradicts your views. Curiosity wanes in the presence of conviction.

Actively seek out information that may contradict your beliefs. If you are a founder, do not do your own research either. There is too much at stake for you not to succumb to this bias.

Confirmation

Instead of changing your beliefs to accommodate new information, you’re doing the exact opposite and adjust your interpretation of the data to fit your beliefs. The higher your IQ, the more prone you are to confirmation bias, i.e. finding a variety of reasons to support your own point of view, selectively looking at evidence and choosing the parts you prefer.

Uncertainty

“No data is clean, but most is useful,” says Dean Abbot. The easiest way to dismiss data is to demand perfection from it, usually on the basis of conviction or confirmation.

Are the time reference and sample group correct? Is the evaluation period long enough? Have I chosen the right measurement method? Am I asking the right questions? What additional questions am I overlooking when I ask this question? Are my assumptions correct? How much confidence do I have in the evidence?

You will not be sure, and psychologically, open-mindedness dies first in the face of uncertainty. There are no easy answers to this problem. Constantly get input from users and benchmark your metrics. Make it a habit to stay current with data analysis. Add context to your data by understanding users’ needs, pains and behaviours. Your confidence and trust in interpreting your data will gradually increase.


Instead of a conclusion, I would like to leave you with a thought from Haruki Murakami’s Pinball:

“Almost nothing can be gained from pinball. The only payoff is a numerical substitution for pride. (…) No, pinball leads nowhere. The only result is a glowing replay light. (…) The goal of pinball is self-transformation, not self-expression. It involves no the expansion of the ego but its dimunition. Not analysis but all-embracing acceptance. If it’s self-expression, ego expansion, or analysis you’re after, the tilt light will exact its unsparing revenge. “

helen ebert Avatar


powered by TinyLetter

Recent: