Diagnosing problems as a product manager — Part I

Something isn’t going right — how do you identify, assess and solve problems that impact your customers?

The key responsibility of a product manager is stakeholder success — i.e. ensuring that customers who use the product benefit from its capabilities, and the team that built the product is made aware of things that can improve the product over time.

Let’s take an example of a mature product, in a post-launch scenario:

Say, as one of the PMs for Lyft (the ride sharing platform), your team notices that ride cancellations have been going up (say, at 15% higher than usual) in the last month. Now, this is obviously not a good thing for the business, and you have to solve this problem.

How do you go about it?

A helpful framework is to go with a ‘broad, then deep’ approach.

Given the open ended nature of the problem, it helps to create a broad set of hypotheses that could be causing the problem, and then tackle each one of these hypotheses individually.

This would help create a comprehensive way to looking at the problem, and a way to rapidly test/validate hypotheses that could be causing it, in addition to analysing factors that could be working together to create the problem.

Let us also clarify the actual problem in place — ride cancellations by drivers are up, (15% over the typical baseline value); and a cancellation is defined by a rider booking a ride, a driver accepting the ride and then the driver cancelling the ride before the pickup is made.

Going back to our example of higher-than-usual ride cancellations, let’s go ahead with this and try to tackle the problem.

List out and test the potential hypotheses that could be driving this outcome

a. Ride cancellations going up may be caused by a gap in measurement

1. Have we seen this scenario before? Is there similar impact on our product ecosystem/other products/services?
2. Has there been a difference in data capture/how we are measuring this metric?

b. Ride cancellations going up may be a repeatable trend

1. What is the time period this has occurred over? Was this a sudden or gradual change?
2. Is there a seasonal impact to this? Is there a characteristic of days (say holidays) where this has happened?

c. Ride cancellations going up may be caused by a certain user segment

1. What is the customer/driver segment most affected?
2. Is there a geography where this problem is isolated?

d. Ride cancellations are going up due to issues software/hardware being used

1. Was there a recent/ongoing deployment or experiment run by us that caused this issue?
2. Is there a certain OS platform/app version where this issue is isolated?
3. Have there been reports of hardware issues encountered/app crashes or recent app updates made?

e. Ride cancellations are happening due to external factors

1. What context of rides are we noticing more cancellations happening (is this around certain location types/peak hours, weather conditions etc.?)
2. Is there an impact from competitor promotions/recent perception of the company that may be causing this behavior?
3. Is this problem noticed more in drivers who drive exclusively for Lyft vs drivers who drive for multiple companies?

Map the user behavior/journey to pin point additional factors that may be causing the outcome

1. Once we have gone with the approach above, listed hypotheses and tested them, we can either isolate the issue, or be left with more questions; let us take a case where we still do not have a conclusive explanation to the problem at hand
2. It could now help to map out the typical driver journey to understand potential pain points that are causing cancellations to go up

-> Driver gets a notification for a ride request

-> Driver accepts the ride

-> Driver sees the rider location/pickup details

-> Driver starts to navigate to the rider

-> Driver either picks up the rider (or) cancels the ride

An analysis to take a set of drivers in a geography where this issue is most prevalent, and divide them into cohorts where we compare drivers who cancel vs drivers who do not cancel, may help isolate the factors that are causing cancellations

1. The reasons we discover could range from drivers who tend to cancel due to getting ride requests from competitors to drivers who cancel due to an inability to find the rider/navigate to their location
2. Some of the questions we ask during this analysis can be:

When is the cancellation happening? Is it at the beginning of pickup, or close to picking up the person?

Is the distance from pickup to destination increasing? Is there something de-incentivizing the driver? Perhaps since the compensation metric is trips/hour, they may prefer shorter pickups

Is there a change in net number of riders/drivers?

Any change in the rider behaviour? Riders are excessively calling drivers? Demographic variations in driver/rider profiles where this problem is observed?

Trying to solve for the problem

1. As seen above, is it important to have a strong validated hypothesis that explains the problem before we get into solutioning; as a product manager, your solutions are a game of trade-offs, and hence the balancing act to be played is key to optimize for outcomes.
2. A few general guidelines to solution can be:

Aligning the problem and its identified root causes to the larger vision/goal of the organisation — this is important when aligning solutions/policies that are consistent with the goals of the organisation; in other words, how do we tradeoff between the driver experience and accomplishing the company vision?

Always run tests/experiments on smaller cohorts/measure their impact before rolling out the solution to a broad base — This is where design thinking, and collaborating with other teams can be useful; rarely is a good solution just a tech-driven one; it does require thinking through the whole value chain, and could range from financial or policy re-planning, design intervention, better training etc.

Based on the findings above, if the problem has been a one-time or rare occurrence (say, this whole thing happened due to a bad software update that auto-cancelled rides); it is important to drive the relevant communications and process changes to ensure tat affected stakeholders are aware and confident of steps being taken to solve the problem

In this series of articles, we will consider more such examples of problems to solve.

As you would have noticed, solving the problem is 80% asking the right questions, and 20% experimenting with potential solutions.