Product Decisions Without A/B Tests: A Causal Brief

The dashboard shows a 12 per cent rise after the new workflow launched.

That sentence contains a measurement. It does not contain a causal answer.

The customers who received the workflow were chosen by account teams. They were larger, more engaged, and already preparing a process change. A seasonal peak began during the rollout.

Any of those conditions could create movement without the product change causing it.

When an A/B test is unavailable, a product manager should not ask an analyst to “prove impact”. They should brief a causal question, expose assumptions, and define a decision that remains safe if the estimate is fragile.

The request that should be refused

A fictional B2B operations product adds automated shift handovers. Account teams offer it first to customers they believe are ready.

Three months later, handover completion is higher in enabled accounts than in accounts without the feature.

The product lead asks:

Can you show that automated handovers increased completion by 12 per cent?

The analyst should refuse the wording.

The observed difference combines treatment, selection, timing, customer mix, and any other change that moved with rollout. The 12 per cent figure cannot be assigned to the feature without further assumptions.

The summary is underspecified even as a description. It does not say whether 12 per cent is a relative change or 12 percentage points, which population it covers, or how completion was defined.

A defensible brief starts with the decision, not the preferred answer.

Exchange one: define the decision

PM: We need to know whether to expand the workflow to all enterprise accounts.

Analyst: What would you do if the effect were positive, negligible, harmful, or too uncertain to identify?

This question prevents analysis from becoming a ceremonial number.

The decision options are:

• expand to a broader but still monitored group;
• keep the current selected rollout while collecting better evidence;
• redesign the workflow because completion improves at the cost of correction work;
• stop expansion because harm or uncertainty is too large.

The analysis does not need to settle the entire strategy. It needs to distinguish among these actions with an agreed tolerance for error.

Data-driven product decisions still require judgement about reversibility, harm, and missing evidence. A method does not choose the risk appetite.

Exchange two: write the causal question

PM: Did the feature improve handover completion?

Analyst: For whom, compared with what, over which period, and which version of the feature?

The team rewrites the question:

Among enterprise accounts eligible in the first rollout window, what would the 14-day completion rate have been if enabled accounts had not received automated handovers, compared with receiving the released version?

This defines:

• population: eligible enterprise accounts in a stated window;
• treatment: access to the released workflow;
• outcome: handovers completed within 14 days;
• counterfactual: the same eligible accounts without access;
• estimand: the average effect for the treated or another named target;
• time: treatment and outcome windows.

The 14-day period is part of the fictional case. A real team should choose a horizon that matches how the workflow creates value.

The counterfactual cannot be observed directly. Causal analysis constructs an estimate under assumptions that must be made visible.

Exchange three: draw how selection happened

Analyst: Why did one account receive the feature and another not?

PM: Account teams chose customers with an engaged operations lead, enough staff, and a planned process review. We also avoided accounts in renewal negotiations.

That answer is more valuable than a sophisticated model with an invented treatment rule.

The team sketches a causal structure:

Account size ───────┬──> Enablement ─────> Completion
                    └────────────────────> Completion

Prior engagement ──┬──> Enablement
                    └────────────────────> Completion

Process review ─────┬──> Enablement
                    └────────────────────> Completion

Renewal risk ───────┬──> Enablement
                    └────────────────────> Completion

Season ────────────────────────────────> Completion

Enablement ──> Workflow use ───────────> Completion

Account size, prior engagement, process review, and renewal risk are plausible confounders because they may affect both enablement and outcome.

Workflow use is a mediator after treatment. Adjusting for it would answer a different question and may remove part of the effect the team wants to estimate.

The graph is a statement of beliefs about the data-generating process. It is not discovered truth.

Microsoft’s DoWhy work stresses explicit causal assumptions and a sequence of modelling, identification, estimation, and refutation. The software can organise reasoning; it cannot supply missing domain knowledge.

Exchange four: inventory evidence before choosing a method

Analyst: Which variables were measured before enablement, and how reliably?

The team makes an inventory.

Candidate variable	Available before treatment?	Measurement concern	Role in the graph
Account size	Yes	Staff count may be stale	Confounder
Prior completion	Yes	Workflow definition changed once	Confounder and baseline outcome
Product engagement	Yes	Multiple possible definitions	Confounder
Planned process review	Partly	Recorded in free-text account notes	Confounder, poorly observed
Renewal negotiation	Yes	Stage changes over time	Confounder
Workflow use	After treatment	Only observed if enabled	Mediator
Operations-lead quality	No dependable measure	Subjective account-team judgement	Unobserved confounder

The missing operations-lead measure matters. Account teams used it to choose treatment, and it may also predict completion.

No regression option makes that problem disappear.

The team can inspect whether a defensible proxy existed before treatment, review the selection process, or narrow the population. It must not manufacture a score from post-treatment outcomes.

Why before and after fails here

A before-and-after comparison asks whether enabled accounts changed after rollout.

It does not show what would have happened to those accounts without the feature.

The planned process reviews could have improved completion on their own. Seasonality could raise handover volume. A measurement definition could change. Customer teams might offer more support during enablement.

Comparing enabled with non-enabled accounts also fails without adjustment because selection was purposeful.

The direction of bias is not guaranteed. Selected accounts may be more capable, making the feature look stronger. Avoiding renewal-risk accounts may remove poor outcomes.

Larger accounts may face more coordination difficulty, pushing the other way.

“We controlled for account size” is not an identification argument.

Exchange five: state the identification claim

Analyst: Under which assumptions can we identify the effect from this data?

A possible claim is:

Conditional on measured pre-treatment engagement, baseline completion, account size, renewal stage, segment, and rollout week, treatment assignment is independent of potential completion outcomes.

That is a strong assumption. The partially recorded process review and unobserved leadership judgement make it doubtful.

The team also needs:

• positivity: comparable accounts must have a real chance of being enabled and not enabled;
• consistency: “enabled” must represent a sufficiently coherent treatment;
• no interference: one account’s enablement should not alter another account’s outcome, unless modelled;
• measurement validity: treatment, confounders, and outcome must mean what the analysis claims;
• time order: adjustment variables must precede treatment where the identification requires it.

If all large, highly engaged accounts were enabled, positivity may fail in that region. The model would extrapolate rather than compare.

Restricting the target population may be more honest than producing an enterprise-wide estimate.

Exchange six: plan challenges to the estimate

PM: Which test will tell us whether the estimate is correct?

Analyst: No single test can do that.

Microsoft researchers make this limit explicit: unlike cross-validation for predictive models, there is no global validator for a causal estimate.

Refutation and sensitivity checks can reveal fragility or inconsistency. Passing them does not prove that every causal assumption holds.

The plan includes:

• compare treatment groups on pre-treatment outcomes and covariates;
• inspect overlap rather than hide unsupported regions;
• use a negative-control outcome that the feature should not plausibly affect;
• run a placebo treatment date before rollout;
• test sensitivity to an unobserved confounder of plausible strength;
• vary defensible outcome definitions and time windows;
• compare more than one suitable estimator;
• inspect whether account-team support changed with treatment;
• repeat on a narrower population with stronger comparability.

A result that changes sign under small, credible assumption changes cannot support a confident expansion.

Hypothesis testing can help structure competing explanations. It should not turn observational evidence into experimental evidence by vocabulary alone.

The result and the bounded decision

Suppose the analysis produces a positive estimate among mid-sized accounts with overlap and stable pre-treatment measurement.

The estimate becomes smaller under sensitivity analysis, and a plausible unobserved leadership factor could explain much of it. Large accounts have almost no untreated comparison group.

There is also a rise in correction work after completed handovers.

The team should not announce that automated handovers increased completion across enterprise customers.

The bounded decision is:

• expand only within the mid-sized account population represented by the data;
• measure correction work as a guardrail;
• randomise or phase later enablement where operationally possible;
• standardise the treatment and account-team support;
• collect the missing selection reasons before considering large accounts.

This decision uses the evidence without asking it to bear more certainty than it can.

If a later controlled experiment is feasible, it can address the treatment comparison more directly. It still needs sound assignment, measurement, exposure, and interpretation.

Engagement analytics can describe who used the workflow and how. Description remains distinct from the causal effect of offering it.

The brief an analyst can challenge

Before work begins, the PM should hand over a brief containing:

1. the product decision and available actions;
2. the population, treatment, comparison, outcome, and time window;
3. how treatment was assigned in practice;
4. the proposed causal graph and its disputed edges;
5. known pre-treatment confounders and measurement gaps;
6. the identification assumptions and unsupported regions;
7. plausible mediators, colliders, and interference;
8. the estimation approaches worth considering;
9. balance, overlap, placebo, negative-control, and sensitivity checks;
10. the decision boundary if evidence is positive, harmful, or inconclusive.

The analyst should be free to reject the brief’s graph, method, or feasible claim. Product context is necessary, not sovereign.

Where to stop

Do not proceed when treatment assignment cannot be reconstructed, critical variables are measured only after treatment, comparable untreated cases do not exist, or the outcome changed definition during rollout without repair.

Do not report a causal number because a stakeholder requires one. Report the descriptive evidence, the identification failure, and the next design that could reduce it.

Observational causal methods can support product decisions. They cannot rescue a question from absent data or make an untestable assumption disappear.

The honest deliverable is often a smaller claim and a safer next move.

Sources

• Microsoft Research: DoWhy causal reasoning project
• Microsoft Research: expressing and validating causal assumptions
• Microsoft Research: DoWhy and the independent PyWhy model

Read next

• From data to decisions: experiment design
• Making smart decisions: data-driven product management
• Research-driven hypothesis testing
• From data to decisions: engagement analytics