Expert product managers, growth managers, and product marketers know the invaluable benefit of measuring customer experience (CX). My research into customers’ interaction with products suggests that product analytics, together with customer interviews and surveys, helps a team make solid decisions.
In this article, I will dive into the first tip from my talk on Measuring and Improving Customer Experience as a PM viz. Product School. I will share my perspective using a recent real-world product experience.
I originally published this in my newsletter on Substack. You can subscribe there to get new articles straight to your inbox.
Why does measuring CX matter?
A comprehensive customer experience plays a pivotal role in helping companies to discern what to correct, improve, or maintain. In times past, millions of successful products have undoubtedly been built by intuition and experience, but data science and analytics now provide us with a better, more accurate way of decision-making. If not you, your competitors are using them to make better product decisions!
Measuring CX to tackle Customer Registration Drop-Offs
Some time ago, I was working with a team to build a KYC registration funnel. When I studied the product analytics, I saw a normal drop-off of customers at each registration stage. However, midway through the registration, analytics showed a sudden 60% drop-off of customers as seen in the chart below. This meant that 60% of users stopped their KYC registration at this stage.
This sudden drop-off of customers was surprising to me. It was even more unexpected when the analytics showed that the drop occurred during a simple step that told customers to wait for verification. After some wait, a congratulatory message was shown to the customers.
To identify the problem, I went through a systematic process. First, I collected quantitative data from product and web analytics; this helped me to identify the problem, prioritize the problem and form some hypotheses. Specifically, the product analytics tool showed us there is a 60% drop so it is a problem to be prioritized. I also hypothesized that the problems could be around the wait.
To further refine my hypothesis, I collected qualitative data from support requests; these qualitative data showed a pattern and allowed me to identify the likely cause of the problem. Specifically, I read a dozen surveys customers wrote when they were done with their KYC registration and another few dozens of customer support requests raised by customers facing issues in the KYC registration.
Then to get more information about this problem and confirm my hypothesis, I did some customer interviews. The interviews revealed that a lot of customers think that the registration step was the last step because we gave them a congratulatory message. Some other customers complained about the wait time of this registration step, they thought they'll get notified after the step was completed, but they were not getting any notification emails.
Arriving at solutions
After I had identified the cause of the customer drop-off, I could address the problem. So what did I do?
1. Customers were notified of the wait time, instead of making them wait without any information or instruction about what happens next.
2. Targeted email reminders were sent to customers that dropped off. These email reminders /notifications were aimed to bring customers back directly to that stage instead of dropping them somewhere else in the web portal.
3. My team and I solved the sequence of steps/stages. We merged the earlier and later parts of the funnel so it was clearer that they have to move forward at each stage of the registration. We also added a clearer call to action at each intermediate step to help them understand that they have to move forward.
4. To avoid confusion, I stopped the congratulatory messages in between the registration flow.
Benefit Of Applying Product Analysis, Surveys, And Interview In Understanding Customer Experience
From the sample experience we used as a case study, we can see the combined effect of product analysis, surveys, and interviews on customer experience. I’ve used this across projects and situations so would like to share the 3 steps that work for me. I’ve covered a few other examples here.
First, the product and web analytics provided a lot of quantitative data, thus notifying us about the sudden drop problem; however, it provides very little explanation for the cause of the problem.
Next is qualitative anecdotes(surveys and customer support data); this provides a lower amount of data; however, it was more accurate and gave me greater insight into the causes of the problem.
Finally, I moved to a personal interview with customers. Since the qualitative and quantitative data had already given me an idea of the cause of the problem, I only asked more targeted questions that helped me understand our customers' mental modes and experiences. Combining these three processes perfectly helped solve the customer experience problem.
I understood from my experience that I can understand and measure customer experience in these three ways. This helped me improve customer experience and business outcomes.
From the pictorial chart above, when you do quantitative data, if you look at the parameter on the side of the inverted parameter, you get a lot of data but very little “why”. The data helps in prioritization, finding patterns, and building hypotheses. From there, you move where the number of data points reduces but the amount of “Why” or empathy increases. So overall, the collection of all of this helps us know the customer experience.
I have summarized this in the table below.
As you would expect, these approaches are essential in retaining existing customers. It is also an excellent way to study a customer's journey and interaction with a product or service directly or via the channel that leads to the product. The combination of these three processes can also serve as one of the most crucial measures to understand whether consumers are staying or departing.
Why not just do customer interviews?
If you were to spend 5 hrs conducting customer interviews vs 5 hrs in one of the 2 other methods, you would get a lot of empathy for customers but no statistical data. You would not know which problems are important to prioritize.
I consider this as the drawback for Google’s HEART framework too. When the framework is used as a data source we assume it is statistically significant although usually it is based on a few dozen users at a time. Instead, the HEART framework should be used as a non-scalable set of metrics.
What is the easiest way to hear from your customers?
If you are worried about crafting a survey and sending it to 100s of customers or concerned about the legwork involved in reaching out to customers for interviews, you can look at support requests from customers or customer reviews of your product. Those are customer-initiated pieces of anecdotal evidence of their likes and dislikes and help you get in the shoes of your customers.
How Merging Product Analytics, Surveys, and Interviews Helps PMs
I like to say that anyone who wants to make better choices needs to comprehend the data and empathetic sides of these combined methods of improving customers' experience. In this way, product managers, for example, can understand their users' actions, make data-driven decisions, measure and perform experiments, enhance activation, conversion, and retention, and create captivating digital experiences. As marketers, they can observe which marketing programs (emails, promotions, social postings, campaigns) bring in the most visitors and which marketing programs bring in the people who are most likely to convert and retain long-term. This information helps marketers direct their efforts and improves product development. By extension, without committing additional technical effort, dev team leaders may eradicate errors, fine-tune functionality, and resolve user friction. I illustrate some of these benefits in this article.
You can watch the video webinar recording for this here.
In a product lifecycle, apart from these methods of measuring experience, there are other methods to improve customer experience. I will share my tips on that in an upcoming post.
Originally published at https://harshalpatil.substack.com on Aug 24, 2022