Measuring Product Success

Photo by Markus Spiske on Unsplash

Hi, my name is Brad. I’m a product manager on the SaaS platform team at SailPoint. Over the years, I’ve interacted with some fantastic people in product management and merged their ideas into many different learnings. I think it’s time that I start sharing them with others. This is the first in a series of posts on these ideas.

To start things off, I want to talk about a concept that I believe is one of the most important for a product manager: measuring the success of what you’ve created. How do you know a feature you’ve created or launched worked out? How do you track OKRs tied to features or even just a simple goal? Regardless of how you monitor your product’s success, it all starts with knowing what to measure and how to take those measurements.

For this post, I’m going to assume that you have a hypothesis for your feature; if you don’t have one, that is ok! I will be using an example one and will write on how to create one in the future. So check back often!

Since I am a platform Product Manager, I will use a platform example, webhooks. It’s a simple concept that many people understand, and it’s common in almost every SaaS application.

Hypothesis

Let’s start with the hypothesis. We’re trying to alleviate the pain that our customers feel managing tedious tasks required for moving data from our product to another SaaS product or automate actions in another app. The customer has more important things to do than manually moving data or kicking off some other app change. Making this a time-consuming pain. We are, therefore, going to build webhooks that can push out important events to third-party systems. Allowing them to create automations to do those tedious tasks. Now we have the customer pain, why the customer has this pain, and an idea for a feature we want to develop. Note, you must have the why the customer has the pain, not just the pain itself. Otherwise, you can miss the bigger picture.

The Feature

In this example, webhooks are a pretty simple concept. You can create a webhook, register it to a set of events, and when something happens in your product, it will then fire off to its chosen destination. Now, you might say initially. Great. We’re going to measure our success by customers creating webhooks. But that’s not the whole picture. That’s just one part. Generally, when you’re trying to measure a product’s success by creating a quantitative goal, you will have multiple conditions to meet to measure success on a per-customer basis.

What to measure

We can begin with an initial condition that the customer has created a webhook. Ask yourself, have they indeed adopted this feature if they made it? They could have created the webhook but not turned it on. We need to add an extra condition, they create a webhook, and they turn it on. Awesome! We now know they’ve made the webhook, and they’ve turned it on, but that’s still not enough. Where’s it going? According to our hypothesis, if the webhook is going nowhere, they’re not getting any value out of it. So we need an extra condition. They create a webhook, turn it on, and configure it to send it to a value destination. In this particular case, let’s say you have a list of possible destinations to value. It might be the domain name contains their company’s domain name or might be to Zapier or Segment. Nevertheless, it’s going to a place of value, not to a test website. We can now successfully measure the customer getting value from the feature and having their pain alleviated.

Setting your target

With this measure, you can confidently say they have adopted the webhook, and we now have a success condition for a customer, but you are a company with a lot of them. Measuring the success of a feature is not going to be determined by a single customer. It’s going to be determined by multiple customers. To make this something that’s like an OKR or just a measurable goal, we need to say that several customers have completed that condition. For the sake of simplicity, let’s pretend we have 1000 customers on our SaaS product. How do you know the number of customers to adopt webhooks to say we have a successful feature? Determining the number of adoptions to call it a success is where things get tricky, but it’s also crucial that you do this right.

Creating this target should build off of your initial product research. You probably have many unique customers, say 100 out of 1000, that have asked about webhooks. This list is a great starting point. You might want to stop there and say, 100 customers in the quarter adopting webhooks are a success. Sadly it is rarely that simple. We might want to ask how complicated is this feature to adopt. For some products, it can be very tricky. Requiring legal approvals to turn it on, code to be injected, or complex authorization with multiple parts. Having assumptions that customers can turn it on a quarter after launch might not be possible, even if they knew the day it launched. We can do some segmentation and find that 70% of our customers are small to mid-market. These customers can move fast and adopt a feature like webhooks quickly. That seems like a reasonable target; out of the 100 customers asking for it, the small and mid-market customers turn it on in one quarter. Note, you want this goal to be something that you just barely hit at the end of your deadline. If you overshoot it, you might not have understood your customers or, more importantly, understood your success metric. Getting it within +- 5% is a great target!

Wrapping it up

To understand feature success, you need two things. The hypothesis that motivates the creation of the feature and a way to evaluate the hypothesis quantitatively. In other words, why does that particular number of adoptions matter? How does this validate or invalidate the hypothesis? If you say 100 and don’t explain why you got 100, then you likely don’t understand the implications of the outcomes. Yet if you try to over-specify your goal measurement or over segment your customers, you can wind up overfitting your problem. Leading to false negatives or false positives. It is a balancing act.

When you do this for all of your features and connect the metrics, it becomes a lot easier to reveal the bigger picture. You can see what the success of any number of features means for your company. More extensive insights come out, and lastly, it becomes a lot easier to drive alignment across your organization.

Whether you’re making an app to track packages or creating a note-taking platform, the fundamentals to measure a product’s success stay the same. Start with the pain, why your solution will solve it, and what it will look like when solved. Then how many customers would benefit from it. Finally, celebrate when you hit your goals!

Platform PM at SailPoint | Open Source Adventure | Ocean Hacker