Experiment Design: Engineer a Comparison You Can Interpret

A collaboration product randomises people to a new approval workflow. One member of a workspace sees the new state, another sees the old one, and both edit the same request.

The dashboard will still produce two columns. The comparison may no longer mean what the team thinks it means.

Experiment design is the work of engineering an interpretable comparison before the feature is built.

It connects the effect a decision needs to the eligible population, assignment unit, real control experience, exposure, outcome, maturity window, and analysis.

If those pieces answer different questions, randomisation cannot rescue the result.

Keep the boundary clear

Product Hypothesis Testing asks which explanation deserves a discriminating test.

Experiment design asks what comparison could identify the effect relevant to the product decision.

Experiment Tracking then preserves the plan, run, assignment, exposure, telemetry, deviations, analysis, and decision that actually occurred.

Do not collapse these practices into one A/B-test template. A precise hypothesis can receive a bad design, and a sound comparison does not validate the wider strategy.

Define the effect before choosing the metric

“Does the new workflow improve approvals?” leaves several effects hidden.

Does the team want the effect of assignment whether or not every member encounters the workflow, or only the effect among requests with a defined opportunity for exposure?

These are different questions and can produce different analysis populations.

Write an effect contract in product language:

Decision: the choice this comparison may change
Treatment: the exact product and service state being introduced
Control: what the same eligible units would otherwise experience
Population: who or what the decision is about
Outcome: the user, business, or operating change of interest
Summary: how treatment and control populations will be compared
Maturity: when the outcome is sufficiently observable
Intervening events: non-delivery, non-exposure, opt-out, switching, collision, or failure
Scope: where the result may and may not be applied

The ICH E9(R1) addendum defines an estimand for a clinical trial through treatment conditions, population, variable, handling of intercurrent events, and a population-level summary.

It is final FDA guidance focused on randomised clinical trials. It says the principles also apply to single-arm trials and observational studies, with greater regulatory interest in confirmatory work.

It is not a product-experiment standard.

Its useful discipline is the order of work: define the treatment effect of interest, then align design, collection, and analysis. Do not let an available estimator decide which question the experiment appears to answer.

For a product experiment, opt-out, delivery failure, switching devices, account migration, and exposure created by treatment can all alter what “the effect” means.

Decide how they belong in the target question before seeing their counts.

Separate four units that teams call “the user”

The unit of intervention receives the product change. The unit of randomisation receives the assignment. The unit of exposure has the defined opportunity to encounter it.

The unit of analysis contributes an observation to the estimate.

They may be the same person. In collaborative, marketplace, device, or enterprise products they often are not.

A workspace can be randomised, a member can encounter the workflow, a request can receive the changed state, and the analysis can report a workspace-level outcome.

Write all four units explicitly and show their relationships:

Randomised workspace
  └─ eligible requests
       ├─ members with possible exposure
       └─ request outcome after the maturity window

Repeated observations from one randomised unit are not automatically independent experimental units. Deng, Lu, and Litz studied variance estimation for online A/B tests with complex randomisation.

Their peer-reviewed 2017 WSDM paper combined theory and simulation with 120 two-week A/A experiments on Skype. The empirical section evaluates variance estimators, not treatment effects.

It shows that naïvely treating randomisation units as independently and identically distributed can underestimate variance under some treatment-effect conditions.

That paper focuses on average-based metrics and large-sample variance estimation. It is not a general certification method for product experiments.

The product lesson is narrower: analysis and power must reflect the assignment mechanism rather than the row count in an events table.

Choose assignment around interference

Individual randomisation is attractive because it can create many assignment units and balance observed and unobserved baseline factors in expectation.

It becomes hard to interpret when one person’s treatment changes another person’s experience or outcome.

Look for interference through:

• shared documents, accounts, queues, inventories, budgets, or rankings;
• messages, invitations, recommendations, or network behaviour;
• administrators configuring states that other people consume;
• sellers and buyers interacting in the same market;
• operational handling across variants;
• treatment that changes who joins or leaves the interaction graph.

Google published a 2018 practice account of designing A/B tests on a Google Cloud Platform collaboration network.

It used the actual network topology and simulation to compare project and connected-component assignment, including stratification.

In that model, component assignment protected treatment consistency and reduced simulated bias from spillovers, while larger and more varied units reduced power.

This was first-party practice research, not a peer-reviewed universal result. The displayed distribution axes were removed for confidentiality, and the conclusions depend on one model and network.

It demonstrates a real design trade-off: a smaller randomisation unit can increase nominal sample size while destroying the isolation needed for the comparison.

Choose among architectures deliberately:

• individual assignment when units do not materially alter one another’s treatment or outcome;
• account, workspace, or cluster assignment when a coherent shared state matters;
• switchback or time-block assignment when the interacting system cannot credibly be split at the same moment;
• another evidence method when no feasible assignment creates the counterfactual the decision needs.

Randomising is not always responsible.

Rights, contractual duties, irreversible migration, or service-continuity risk can make a controlled rollout, simulation, audit, or observational design more appropriate.

Make control a real product state

Control is not “nothing.” It is the experience the eligible unit would receive without the treatment during the experiment.

Name its release, configuration, service behaviour, policy, and concurrent changes. If control receives an unrelated redesign, the alternative has changed.

Treatment also needs a versioned contract:

• visible product and content differences;
• backend, model, ranking, latency, and policy differences;
• delivery and eligibility rules;
• dependencies and fallbacks;
• failure modes that create a third experience;
• what must remain identical across variants.

Test assignment and delivery separately. A stored treatment flag does not prove the intended experience arrived.

If a partially configured treatment is a plausible production state, decide whether it belongs in the effect or invalidates the run.

Do not condition on behaviour treatment can cause

Teams often analyse “people who used the feature.” That group exists only after treatment and can select for motivation, opportunity, device, permissions, or a successful delivery path.

Comparing exposed treatment users with all control users breaks the original randomised comparison.

Triggered analysis can be valid under a design that defines a corresponding trigger for both assignments.

Deng and Hu’s peer-reviewed 2015 WSDM paper derives exact dilution formulas and a variance-reduction approach for low-coverage features where triggering can be identified in treatment and, through counterfactual logging, in control.

It also shows that translating a triggered effect to the full population depends on metric form and assumptions.

The paper does not justify filtering on an event observable only after treatment or make post-hoc exposed-user comparisons causal.

Define before launch:

• the trigger and whether both variants can satisfy it under comparable conditions;
• assignment, delivery, and exposure populations;
• the primary population for the effect;
• how non-delivery and non-exposure enter the question;
• whether an exposed-population analysis is diagnostic or decision-bearing.

Preserving assignment may estimate the effect of offering the change, including delivery and adoption friction. That can be the product effect leadership needs.

Build an outcome stack around the decision

An experiment should not have fifteen co-equal success metrics.

Use distinct roles.

Primary outcome

Choose one decision-bearing outcome close to the intended value and mature within the study horizon. Define its population, denominator, identity, time, and metric version.

Mechanism diagnostics

These show whether the expected path occurred without silently becoming alternative primary outcomes.

Guardrails

Choose plausible displacement or harm created by the treatment and state which movement requires pausing, investigation, or rejection.

Telemetry-health checks

Assignment balance, join coverage, missing events, duplicate records, release parity, and schema health concern whether evidence is trustworthy.

They are not product outcomes.

Conversion Tracking covers the underlying event, identity, state, and reconciliation contract.

An experiment references those definitions; it should not recreate them in a notebook.

Power the decision-relevant effect

“How many users do we need?” cannot be answered before the team names the smallest effect that would change its decision.

A detectable movement smaller than the cost, risk, or opportunity threshold may be statistically interesting and operationally irrelevant.

Power depends on more than traffic:

• baseline variation and event rate;
• smallest decision-relevant effect;
• number of independent assignment units;
• allocation across variants;
• clustering and repeated observations;
• outcome maturity and censoring;
• planned comparisons and analysis method;
• missing data, non-delivery, and expected attrition from the evidence path.

Do not copy a standard sample size or “two-week minimum.” A renewal outcome may need months; a severe technical guardrail may need minutes. Novelty has no universal expiry date.

If the feasible design cannot distinguish effects large enough to change the decision, do not run an underpowered ritual and call a flat result “no impact.”

Narrow the population, choose a more sensitive valid outcome, reduce variance through a prespecified method, test a larger contrast, extend the horizon where responsible, or use another form of evidence.

Treat time as part of the design

Switchback designs assign treatment across time intervals rather than maintaining simultaneous treatment and control groups.

They can be useful when marketplace, logistics, ranking, or service units interact too strongly for concurrent individual assignment.

They also introduce carryover: treatment in one interval can affect outcomes after the system switches.

Bojinov, Simchi-Levi, and Zhao’s peer-reviewed Management Science paper derives optimal switchback designs and two inference approaches under assumptions about the order, or duration, of carryover.

It is a theoretical design paper with simulations, not evidence that switchbacks fit every marketplace or that one washout duration is safe.

For a time-based design, specify:

• interval and randomisation schedule;
• expected carryover and washout handling;
• relevant demand and operating cycles;
• incidents and concurrent releases;
• which system state must be restored between assignments;
• the analysis that matches the schedule.

If treatment changes future inventory, learning, network membership, or operator behaviour, switching the flag may not switch the system back.

Fix analysis and stopping before outcome data argues back

Specify the primary comparison, estimator family, uncertainty treatment, exclusions, planned segments, and handling of missing or corrupted evidence before reading the result.

Choose a fixed-horizon design or a valid sequential method in advance. Repeatedly checking an ordinary fixed-horizon result and stopping when it becomes attractive changes the procedure.

Exploration remains valuable. Label post-hoc segments and alternative outcomes as exploratory, preserve how they were found, and test consequential findings independently where the decision warrants it.

Write stop rules for two different jobs:

• operational stops: harm, security, privacy, broken delivery, or service failure;
• inferential stops: evidence conditions defined by the statistical design.

An emergency stop protects people and systems. It does not automatically make the partial effect estimate decision-ready.

Run a design review before implementation

A useful review tries to break the comparison, not approve the template.

Ask:

1. Which exact effect would the proposed result estimate?
2. Can a relevant unit affect another unit’s treatment or outcome?
3. Do intervention, assignment, exposure, and analysis units align?
4. Does control represent the real alternative throughout the run?
5. Can treatment change eligibility, triggering, exposure, or measurement?
6. Are primary, diagnostic, guardrail, and telemetry measures separate?
7. Is the decision-relevant effect feasible to resolve under clustering and maturity?
8. Do analysis and stopping rules match the assignment design?
9. Could either result change the stated product decision?
10. Is a randomised experiment responsible and proportionate here?

Fabijan and co-authors studied how experts inspect online controlled experiments across nine companies, each running hundreds or thousands of experiments per year.

Their peer-reviewed 2019 ICSE practice paper reports that much analysis work happens before launch. It does not prove that a checklist causes trustworthy decisions.

A fictional workspace experiment

Return to the fictional approval product.

The decision is whether to make a redesigned approval state the default for multi-member workspaces.

Individual assignment would give the team more rows, but members edit the same requests and administrators configure permissions for everyone.

Treatment can therefore alter a control member’s state and behaviour.

The team chooses workspace assignment. The design has fewer independent units, so its analysis and power calculation must reflect clustering.

Treatment and control release states are frozen. The exposure definition is diagnostic; the primary comparison retains all eligible assigned workspaces.

The primary outcome represents decision-ready reviews. Diagnostics cover rendering and role comprehension; guardrails cover permission errors and support contact.

No result is supplied. The design may prove infeasible, showing that the available A/B test cannot answer the decision responsibly.

Freeze the comparison, then hand it to execution

Before launch, retain:

1. decision, effect contract, and scope;
2. treatment and real control versions;
3. intervention, randomisation, exposure, and analysis units;
4. interference and carryover assumptions;
5. eligibility, trigger, delivery, and exposure definitions;
6. primary outcome, diagnostics, guardrails, and telemetry checks;
7. decision-relevant effect, power assumptions, and feasibility;
8. analysis, segment, multiplicity, and stopping plan;
9. risk routes, authority, and invalidation conditions.

Then version the design and hand it to the experiment record.

The purpose is not to predict every implementation problem. It is to ensure the comparison that reaches the dashboard still answers the question the product decision asked.

Sources

• FDA: ICH E9(R1), Estimands and Sensitivity Analysis in Clinical Trials (final Level 1 guidance issued in May 2021, focused on randomised clinical trials but also applicable to single-arm trials and observational studies; this article borrows only the distinction between an effect target and its estimator)
• Yoon: Designing A/B tests in a collaboration network (first-party 2018 Google Data Science blog article using the actual GCP collaboration graph and simulation; not peer reviewed or a universal assignment rule)
• Deng, Lu, and Litz: Trustworthy Analysis of Online A/B Tests (peer-reviewed WSDM 2017 methodological paper on variance estimation for average-based metrics; combines theory and simulation with 120 Skype A/A experiments)
• Deng and Hu: Diluted Treatment Effect Estimation for Trigger Analysis (peer-reviewed WSDM 2015 methodological paper deriving dilution and variance-reduction methods for low-coverage triggered experiments)
• Bojinov, Simchi-Levi, and Zhao: Design and Analysis of Switchback Experiments (peer-reviewed theoretical Management Science paper on optimal switchback design, carryover-order assumptions, inference, misspecification, and simulations)
• Fabijan et al.: Three Key Checklists and Remedies for Trustworthy Analysis of Online Controlled Experiments at Scale (peer-reviewed ICSE-SEIP 2019 practice paper synthesising expert experiment-analysis checks across nine high-volume programmes)

Read next

• Product Hypothesis Testing: Make Explanations Compete

• Experiment Tracking: Reconstruct Every Result and Decision

• Conversion Tracking: Build a Trustworthy Measurement Contract

• Guardrail Metrics for Product Decisions