On metrics and measuring success

✨ Hi, I’m Sara Tortoli and this is the May edition of The Plunge Club, a monthly newsletter dedicated to product and human tinkering.

If you are not a subscriber yet, then take the Plunge and join the Club:

Demystifying KPIs

I am a big believer that the type and quality of questions I ask myself are going to determine the quality of the life I am going to live. After all this newsletter, always ends with the "question I am asking myself" section.

I make a point to come up with new, better questions to ask myself and usually leave my mind to mull over the answer, for however long it takes.

The same principle can be applied to a product, and it is especially important when it comes to selecting KPIs. Choosing and managing the correct data and Key Performance Indicators (KPIs) for a product is one of the most important responsibilities of my job as a product manager. With so much data available however, the problem is to choose the right KPIs to focus on.

I used to approach KPIs as some sort of entity with a life of its own, getting overly focused on frameworks and methods. To be clear, I have nothing against frameworks. I still use the AARRR framework and others to help me out. Over time however, I learned that metrics are just a means to monitor and find an answer to questions I wonder about the product and its users.

So rather than focusing on a metric or on a framework per se, I started to focus on questions and ways to answer these questions.

Understanding the context and bigger picture

Before setting any KPI, it’s important to understand the context and the type of product we are dealing with. There are three elements I consider:

• product stage or maturity

• business model

• company objectives and strategy

Depending on these factors, the questions I ask myself are going to differ, as is the way (or the ability) to get the answers.

For example, consider the different stages of maturity of a product. Pre-launch, when you don’t have any data, you rely mostly on qualitative feedback. Other factors to consider are market data, companies in a similar market, and potential feedbacks on prototypes. After a product launch, the data floodgates open. Suddenly you can get direct feedback and see how new marketing or features launch affect KPIs.

The business model of the company also heavily influences the types of metrics to choose from, as in the example below.

Even considering the context and type of product, the risk of getting overwhelmed by the amount of data is high.

This is why a third element to consider is the business objectives and strategy of the company. What is the ultimate vision? How can we measure if we are on track towards its fulfillment? Considering both long term and short objectives of the company is crucial to come up with the right metrics to track and filter out data that are not relevant. One easy way to accomplish this is by looking at the company OKR. I have previously written about OKR and the difference between OKR and KPIs here.

Measuring success: how to come up with the right questions

By looking at product stages, business model, and business strategy and objectives (OKRs), we should have an idea of what success looks like. This guides us in understanding what is important and what’s not.

The next step to narrow our focus is to consider the product itself and write down all the questions that are relevant to measure its success. The most common questions are reported below, however those vary greatly depending on context:

• How satisfied a customer is with a product or feature?

• How often users engage with a product or feature in a given timeframe?

• How many new users are we onboarding?

• Where and why users drop out of the funnel?

• How many customers come back to the product?

• How easy a product is for customers to use?

• What is the financial impact of the product?

These questions can now be split and turned into metrics or KPIs. Before picking up a specific metric however, there are other preliminary questions to consider, specifically:

• Can this metric be easily quantified (aka can you calculate it and if so how)?

• Are we able to influence/drive change using this KPI, or is it out of our control?

• Does this KPI connect to our objective and the overall strategy?

• Is it simple to define and understand?

• Can we measure it on time and accurately?

• Does it contribute to a broad range of perspectives (e.g. customers, operations, finance, marketing)?

• Will it still be relevant in the future?

If a metric doesn’t fulfill one or more of the above questions, chances are it is not a good fit, at least for the time being. If on the other hand it meets the criteria, you can further define:

1. How to calculate the metric

2. The type of metric. If you are using the AARRR framework for example write down which step of the funnel this metric belongs to.

3. How often you need to look at this metric (daily/weekly/monthly/quarterly)

4. Through which tool can you get this KPI? For example Google Studio, Tableau…

5. Who are the main consumers of this KPI that need to be informed? Most often the main consumers are also responsible for taking action if needed.

Below is a table with an example, based on a metric that we recently set after launching a chatbot to help job seekers find a job. One of our key metrics is the feedback on intent recognition or intent score.

Limitations of KPIs

The effort to find an answer to questions we have on the product through data is not always rewarded. This frustratingly happens more often than not.

Data can go only so far, they signal possible issues, but not why. You are left to figure that out on your own. This especially trues if you release too many changes at once at the end of a development cycle. How can you reconduct a shift in behavior or usage to a specific cause if you deploy multiple updates in one setting?

The more the product matures, the more data you have available, the more tempting is to forgo direct contact with users. Over time however, the lack of “why” that comes from only looking at data, makes you drift away from your users.

This is why looking at the data and KPIs is only the first step. When there are signals of issues happening through various touchpoints or steps of the funnel of the product, it’s important to get to users as fast as possible to understand why. This can be done through various means, from user testings to user interviews.

Once we get to the root of why something is happening, we can take appropriate action to correct it.

The “Now” section

🎧 What I am listening: Adam Robinson talking on the Tim Ferris show about struggling with meditation and developing attention and awareness of others

📚 What I am reading: Seeking Wisdom: From Darwin to Munger a collection of thoughts on how our brain works and how we can be aware and take advantage of our own limitations.

🥁 What I am doing: I am taking some time off from my usually busy Berlin life and enjoying family time, as well as some much-needed sun and nature in the heart of Tuscany. Lesson learned: it’s important to give yourself time away from your everyday reality, even when you don’t think you need it. You start to see things differently.

🧐 Question I am asking myself:

This a question I started to ask myself whenever I have to make a decision or approach a new situation.