Some time ago, My team and I launched a feature in our product to provide self-service capabilities to customers and eliminate the biggest reason they created support requests. However, even after several months, there was no reduction in customer support tickets. Is our product feature not working? Or is there something else going on that we do not know?
In this article, I will use this real-world example to explain the importance of using metrics and signals to get leading customer experience measures of your releases. In an earlier article, I discussed merging product analytics, surveys, and customer interviews to do effective customer.
I originally published this in my newsletter on Substack. You can subscribe there to get new articles straight to your inbox.
Launching a new product or feature can often be tricky for Product Managers. Despite customer research and good execution, there is still much uncertainty about your customer’s reaction to the launch.
I will dive into the second tip from my talk on Measuring and Improving Customer Experience as a PM at Userpilot’s Product Drive Summit. Here, I will share my perspective on another best practice you can implement to improve customer experience using a recent real-world product experience.
Thumbnail credits to Image by storyset on Freepik.
Output Metrics vs Input Metrics
Metrics are quantitative data used to monitor, evaluate and compare various aspects of a product. They tell you how customers interact with your products and how that interaction affects your business.
There are generally two product performance metrics you should be concerned about as a product manager: Output metrics and Input Metrics.
Output metrics are your end goal as a product manager. They are the eventual results of your product launch; examples are revenue, the number of paying consumers, the NPS, and customer retention. These measures show how your business is performing.
Input metrics, however, refer to the effort put into a business venture. As a product manager, they refer to the effort you put into ensuring a successful product lunch. Input metrics are generally under your control. They include blog posts, page latency, sales training hours, support shortcuts, sales calls, building customer-requested features, or the time it takes for the first response to a support inquiry.
What are Signals?
Signals are the early indicators that tell you if your product lunch is going according to plan or not. They are essential to product managers as they allow you to identify the problems with your current strategy and make improvements as early as possible.
For example, receiving positive customer reviews about a new product is a good sign that you are on the right path. You know your strategy is working, and you can continue its implementation.
However, if you suddenly receive a lot of complaints or negative comments, this indicates that you have done something wrong and should re-evaluate your strategy.
Why Didn’t the Feature Launch Move the Metrics?
There was a time my team was getting hundreds of support requests to change billing settings for customers. This indicated an unhappy customer experience. So, we set our target to reduce the number of support tickets. To improve customer experience, we decided to launch a new product feature.
We built a self-service feature where customers can visit the web portal to update and save their payment address, purchase order number, and email.
However, several months after our new product feature launch, I observed no significant reduction in the number of customer support tickets. Is our product feature not working? Or is there something else going on that we do not know?
Identifying Signals, a.k.a. Leading Indicators of Success
How do I know whether the feature is not working or if something else is wrong? I did not have web analytics, user reviews, or see any support ticket reduction. I could only see engineering metrics. I looked at the number of API calls from the feature in the web portal. Surprise! it seemed that users were using it.
So, although the support tickets were high, there was a latency and benefit from launch time. To know whether I was directionally going in the right direction, I needed to identify early indicators.
In this case,
The desired change or the Output Metric was the normalized number of support requests from customers
The Input Metric was launching feature(s) which was entirely in control of my team
Signals were between the Input and Output metrics, ranging from the number of views of FAQ docs to webpage views.
Putting this on a time continuum of the product lifecycle, we can see that output metrics change much later, while input metrics are much more in your control and influence the intermediate signals and output metrics.
3 Takeaways for Measuring Success of your Product Launch
From this case study, I would like to highlight 3 takeaways for measuring the success of your product launch.
1) Track the success of your launch using both input and output metrics, not just output metrics.
2) You can control input metrics - e.g. launch of a product. Find input metrics since they help measure your efforts. It would be best if you got buy-in from your leadership that the input metrics identified will cause the output metrics to move in the desired direction.
3) Find signals or early indicators of progress. What are some sample signals? Let’s take a look below.
Examples of Signals: A Timeline of Metrics After Product Launch.
Not sure what are some examples of Signals? Here is a sample timeline of metrics. The ones to the left are “input metrics”, which your team can control, whereas the eventual business outcomes you are looking for at the rightmost ones - “output metrics”.
Your level of control on these metrics reduces as you move from left to right.
I have a few FAQs in the rest of the article below.
What if you are Evaluated based on Output Metrics only?
I’ve come across businesses or product teams which are held accountable for output metrics such as revenue, profits, or the number of customers. This accountability is not ideal since not just Product/Engineering but other functions such as sales and factors outside the business influence those metrics.
Without getting into “change the culture of your organization”, I would suggest bringing in signals to the discussion as metrics you will measure and showcase in your product launch dashboard, in addition to Output Metrics. Adding Signals to your dashboard will show your progress without a sudden change. It can gradually change stakeholder expectations.
What is the delay you can expect for Output Metrics?
Although this will vary based on the product type, the scope of the feature, and your customer evangelization efforts, some examples:
6 months+ for SEO efforts to improve revenue
3 months+ for a UI change to improve NPS
What if you do not have visibility to Signals?
I could not access webpage views, FAQ document views, drop-offs, or sales calls in the case study above. Similarly, I understand that sometimes you may not have access to data points.
Go back to why did you decide to prioritize this feature? What data points told you this is a big problem?
How did you decide on what the problem and solution are? What qualitative and quantitative data helped you?
Those would be some ways to find numbers accessible to you for measuring the success of your product launch.
How are Signals/Early Indicators Important to Product Managers?
As a product manager, one of your most essential skills is quickly detecting how well your customers are accepting your new products. Keeping track of input and output metrics is vital as they ultimately tell you if your product lunch was a success or not in improving the customer experience.
However, it is easy to get lost in analysing your output metrics. A significant factor that product managers often overlook is the analysis of early indicators. Analysing early indicators (Signals) is very important as they give you an early insight into the mood of your customers about a new change, and they allow you to make adjustments as early as possible.
Where Can I Watch This?
You can watch the video webinar recording for this here.
Apart from identifying success measures in a product lifecycle, there are other methods to improve customer experience. I will share my tips on that in an upcoming post.
Originally published at https://harshalpatil.substack.com on Oct 5, 2022