The product manager interview playbook — Part III: Analytics questions

5 min readMar 13, 2022

These questions test your ability to reason about metrics, data and strategy.

Let’s start with metrics.

If an interviewer were to ask you — “What are some key metrics that YouTube would be keen to track, analyse and optimise?

You could say:

Youtube has a goal of driving user engagement (i.e. through content consumption), so that they can both provide value to the user, as well as drive revenue through ads or premium subscriptions.
But engagement is also a function of user experience; i.e. users need to have a seamless experience with the platform in terms of load times, actually finding relevant videos (if search is used) and not being bombarded with too many ads.
Bad user experience = low engagement = reduced top-line; subsequently impacting the other stakeholders in the ecosystem, which are advertisers and creators.

All right, so say we have some idea about what the goals are. What actions from the user would help YouTube achieve their goals?

Engagement can, of course be a function of multiple things; we’d want our users to maximise things such as — time spent per session, # of videos watched, # of recommended videos watched, average time watched per video, # likes, # comments, # shares and so on.

Metrics:

Now that we defined our goals and expected actions from users, metrics can be chosen. It is key that metrics always align with the overall goals and product vision, and are able to effectively measure user actions/behavior.
Metrics also depend on the context — our goals here are defined at a very uber level; it helps to get down to a specific case — Eg. Videos in a certain category/region are experiencing high drop off rates — what metrics will help us understand why this is happening?

Evaluate:

Analysing metrics requires good analytical thinking — the ability to study data and obtain insights into WHY something is happening, and WHAT can be done to improve the outcome
Interestingly, over optimising metrics can also cause negative behavior; example: optimising for maximum comments may also lead to negative/hateful/spam comments which impact user experience; similarly, over optimising for watch time may also be negative in the long term for users

So, if we to generalise such questions into a framework:

G — Define goals and overall product vision

A — Analyse what actions/user behavior will achieve goals

M — Define metrics that will help us measure actions/user behavior so that it can be optimised

E — Evaluate your metrics; test for false positives/negatives and apply an analytical process to inform your roadmap

And you’ve got GAME.

We just applied this to the context of Youtube’s example above.

Frameworks are useful, but as a guide, not as a go-to answer all the time.

Let’s look at a few questions that are a bit more varied.

What are some metrics or A/B tests we can run to improve Google’s homepage?

An approach to such questions can be to go with hypothesis based scenarios
What elements on the Google homepage currently exist? (Main search bar, suggestive search, language options, voice based search, footer links, image search, links to other Google products/services)
What is our goal? i.e. what is the intended outcome of this improvement? Say, we want to make users adopt voice search over text based search (hypothetically)
We can now look at metrics in terms of search split trends, and slice the data by different dimensions (looking at region, user cohorts and demographics etc.); this can help us identify a cohort that can possibly be influenced to use voice search more — either by solving for awareness, ease of search or the general UX.
Based on the analysis above, a limited set of users can get exposed to an alternate page design or user flow.

These are high level ideas that may help nudge you in the right direction, when asked such questions

What are some metrics we can study to improve the customer experience with Uber?

Let’s define a sub-goal here — we want to optimize for the pickup ETA (i.e. the time elapsed from ride acceptance by driver to customer pickup).
We can have a couple of levers here:
Optimising for location identification and routing — analysing wrong turn rate/bad directions, issues in locating the passenger etc.
Optimising for user experience — providing accurate wait time estimates, a choice of how the driver and customer may engage to ensure a smooth pickup process

A similar process can be followed to derive a meaningful approach, from the examples above

Overall, analytics would be key in the following respects:

Metrics to track → How to measure chosen metrics effectively → How to communicate the results effectively
Products need to be setup with instrumentation/tools to track and measure metrics that are relevant
Dashboards/reports needs to be contextualised to the audience and give them insights to improve the product over time
This would lead to the product being data driven, fostering a culture of experimentation, A/B testing, cohort analysis etc.

Choosing the right KPIs → This is key to ensuring the product is evaluated in the right way

KPIs should be selected based on their relative impact to customers and the business
They should be responsive, but not volatile
Better to choose leading indicators than lagging ones

As an example, here are some KPIs by type of product:

A consumer email app

An online advertising platform

Monthly run rate compared to the same month prior year (to account for seasonality)
Advertiser cost per click (CPC) and cost per acquisition (CPA)
Advertiser retention, upsell, downsell, and churn (same advertiser versus previous month)
Advertiser satisfaction scores and net promoter scores (surveyed quarterly)

An internal infrastructure tool

An email campaign for an e-commerce website

SaaS company with freemium model

We will tackle more aspects of the PM interview, such as execution and product questions, in the next set of articles :)

Written by Thomas Varghese