Making Smart Decisions: A Practical Guide to Data-Informed Product Management

A dashboard can be perfectly accurate and completely useless.

It may show 40 metrics, update every minute, and still leave the team arguing about what to build. The missing ingredient is rarely more data. It is a clear decision that the evidence is meant to inform.

That is why I prefer data-informed to data-driven product management. Data should challenge judgement and improve it. It should not pretend to replace context, responsibility, or choice.

Start with the decision, not the dashboard

Before opening an analytics tool, write down the decision in plain language.

“Improve onboarding” is not a decision. “Decide whether to simplify account setup or improve the first project flow” is.

A useful decision statement contains four parts:

• the choice the team needs to make;
• the outcome it wants to influence;
• the population affected by the choice;
• the date or condition that triggers the decision.

Now ask what evidence could change the answer. If no plausible result would alter the plan, the analysis is decoration.

This test saves teams from collecting data to defend a decision already made. It also exposes decisions that are really matters of principle, risk appetite, or strategy.

For the operating system behind this work, see Building a Product Metrics Practice.

Choose evidence that represents the outcome

Metrics are proxies. A click is not satisfaction. Time in an app is not automatically value. A conversion can improve while the customer experience gets worse.

Start with the outcome, then build a small measurement chain:

Layer	Question	Example
Outcome	What should improve for the user or business?	More teams reach a useful first result
Behaviour	What observable action suggests progress?	A team completes its first workflow
Diagnostic	What helps explain the result?	Setup errors by step
Guardrail	What must not deteriorate?	Support requests or early cancellations

Google’s HEART framework offers five user-centred dimensions: happiness, engagement, adoption, retention, and task success. Its value is not the acronym; it is the move from goals to signals and then metrics.

Choose the smallest set that can support the decision. A metric deserves a place when someone can explain how a change in it would alter an action.

Segment before averaging. An apparently stable result may hide a serious decline for new customers or a strong improvement for one use case.

Check whether the data deserves your trust

Teams often debate meaning before checking whether the event was captured correctly. That reverses the order of work.

Before interpreting a result, check:

• Is the event definition unambiguous?
• Did the implementation change during the period?
• Are internal users, bots, or retries included?
• Is the population complete enough for this decision?
• Are time zones and reporting windows consistent?
• Can the number be reconciled with another reliable source?

A sudden improvement after a tracking release is not yet a product insight. It is an invitation to inspect the instrumentation.

Document metric definitions beside the analysis. “Activated user” should mean the same thing in the dashboard, the experiment, and the quarterly review.

If the definition changes, preserve the history. Otherwise, the chart may look continuous while comparing two different behaviours.

Combine behaviour with explanation

Quantitative data tells you what happened at scale. It rarely tells you why it happened or what the experience meant to the person involved.

Qualitative evidence provides that missing context. Interviews, support conversations, session reviews, and usability tests can reveal confusion, workarounds, competing priorities, and language the event stream cannot see.

The two forms of evidence work best in a loop:

1. Behavioural data reveals a pattern worth investigating.
2. Research suggests plausible explanations.
3. The team turns those explanations into testable changes.
4. Measurement shows whether the change affected the intended outcome.

Suppose completion drops at the final onboarding step. The funnel locates the break. Interviews may reveal that users do not have the requested information yet.

The right response might be to postpone the question, not redesign the button. Without context, the team could optimise the wrong part of the experience.

Turn an insight into a decision

“Users who invite a colleague retain better” is an observation. It is not yet a recommendation.

The relationship could mean that invitations create value. It could also mean that already-committed teams are more likely to invite colleagues. Correlation alone cannot tell you which story is true.

Write an evidence note that keeps the layers separate:

• Observation: What does the data show?
• Interpretation: What might explain it?
• Uncertainty: What else could be true?
• Decision: What will we do now?
• Learning plan: What result would confirm or challenge the choice?

This structure makes disagreement productive. A colleague can accept the observation while questioning the interpretation, instead of arguing about “the data” as one indivisible object.

When causality matters, design an experiment proportionate to the decision. Microsoft’s published work shows the organisational value of controlled online tests.

Not every product has the traffic or conditions to run one well. In that case, use the strongest feasible method and be candid about what it cannot prove.

For a practical path from question to test, read From Data to Decisions: Experiment Design.

Build a repeatable decision cadence

Good analysis loses value when it arrives after the decision or disappears in a slide deck.

Create a cadence that joins measurement to action:

Weekly: inspect material changes, data quality, and immediate risks.

Per initiative: review the outcome, leading evidence, guardrails, and next decision.

Monthly or quarterly: challenge whether the metrics still represent the strategy and retire measures that no longer guide action.

Give each important metric an owner. Ownership does not mean manipulating the number. It means maintaining its definition, quality, context, and connection to a decision.

Keep a decision log. Record the evidence available, the choice made, the confidence level, and the expected result. Review it later.

The point is not to score who was right. It is to improve how the team reasons under uncertainty.

Use data without quietly expanding its purpose

Responsible data practice starts before a consent banner. It starts when the team decides what it genuinely needs to collect.

The UK Information Commissioner’s Office describes purpose limitation as specifying why personal data is collected, documenting that purpose, and avoiding incompatible reuse without appropriate justification.

For a product team, that leads to practical questions:

• Can we make this decision with less data?
• Does the person understand the purpose of collection?
• Are we retaining the data longer than the purpose requires?
• Could this segmentation create unfair treatment or expose sensitive traits?
• Who can access raw data, and why?

Do not collect an event because it may become useful someday. Undefined future value is a poor trade for permanent privacy and security risk.

Aggregate where individual detail is unnecessary. Restrict access. Involve privacy and legal specialists early when the decision affects personal data or automated treatment.

Ethical use is not a constraint added after analysis. It is part of deciding whether the evidence is fit for use.

Working artefact: the Decision Evidence Brief

Complete this brief before producing the final dashboard or recommendation. A blank field is a research gap, not an invitation to invent certainty.

Field	Decision-ready entry
Choice	The alternatives and the person accountable for choosing
Intended outcome	Behaviour or condition that should change for a defined population
Decision signal	Metric, definition, window, and why it represents the outcome
Data fitness	Instrumentation checks, exclusions, missingness, and known bias
Explanatory context	Research or operational evidence that explains observed behaviour
Counterevidence	A fact or segment that weakens the preferred interpretation
Guardrails	Outcomes that must not deteriorate
Action	Proceed, reshape, research, defer, or stop
Reconsideration trigger	Result, date, or condition that reopens the choice

This layout keeps measurement and judgement in the same record. It also shows reviewers where the evidence stops and responsibility begins.

A decision-ready checklist

Before presenting an analysis, ask:

• Is the decision explicit?
• Can the evidence realistically change it?
• Does the metric represent the intended outcome?
• Have we checked instrumentation and definitions?
• Did we inspect meaningful segments rather than one average?
• Have we separated observation from interpretation?
• What qualitative context is missing?
• What action follows, and what would make us reconsider it?
• Is the collection and use of data proportionate to its purpose?

Data-informed product management is not a contest to produce the most charts. It is a practice of making uncertainty visible, gathering evidence that matters, and remaining accountable for the choice that follows.

The team still has to decide. The data helps it do so with fewer convenient stories and better questions.

Sources

• Measuring the User Experience on a Large Scale: User-Centered Metrics for Web Applications — Google Research
• Online Experimentation at Microsoft — Microsoft Research
• Purpose limitation — UK Information Commissioner’s Office

Read next

Building a Product Metrics Practice shows how to make definitions, ownership, review, and learning part of the team’s operating rhythm.

• When Product Research and Analytics Disagree