Experiment Tracking: Reconstruct Every Result and Decision

An experiment can have a result and still lack a defensible history.

The chart may name the variants and report an effect. It may not show which plan was active, who qualified, what was assigned, which build delivered the treatment, or whether missing telemetry differed by variant.

Six months later, the number remains visible while the conditions that produced it have disappeared.

Experiment tracking should preserve the chain from an intended test to an accountable product decision. Its quality test is reconstructability, not the number of rows in an experiment catalogue.

This is not a guide to writing hypotheses or choosing a statistical design. It does not define product events, interpret engagement, or replace the analyst’s work with a dashboard.

It gives those activities durable provenance: what existed, which version was used, what changed, what evidence was produced, and what the organisation decided.

Keep the intent narrower than experimentation

Five related practices need distinct ownership.

Hypothesis work states the uncertain mechanism and the claim that could be challenged.

Experiment design chooses the intervention, comparison, unit, estimand, method, outcomes, power assumptions and stopping approach.

Conversion and event tracking defines what observed states mean and whether their capture is trustworthy.

Engagement analytics interprets behaviour against a product’s value model, population and cadence.

Experiment tracking connects the exact versions of all four to a run, an analysis, a decision and later follow-up.

A dashboard is one view of that record. If changing a filter can silently change the population or metric version, it is not the record itself.

Give the experiment a stable identity and immutable history

Use one durable experiment identifier for the question being investigated. Give each materially different execution its own run identifier.

A run changes when the evidence can no longer be interpreted under the same operating conditions. Examples include a new randomisation unit, eligibility rule, treatment build, allocation method or primary analysis.

Do not overwrite the earlier run to make the page look tidy. Link the runs as revisions or successors and explain why the boundary moved.

Before the first assignment, freeze a versioned plan containing:

• experiment and run identifiers;
• question and decision owner;
• design and analysis-plan versions;
• eligibility definition and randomisation unit;
• expected allocation and assignment method;
• treatment, control and release identifiers;
• exposure definition;
• primary outcome, guardrails and metric-contract versions;
• telemetry-health checks;
• planned start, review and stopping conditions;
• approved data purpose, access and retention;
• owners for execution, analysis and incident response.

The live record can continue to grow. The frozen version must remain visible beside later amendments.

OSF defines preregistration as a time-stamped, read-only study plan posted before data collection or analysis. Its official guidance also asks researchers to anticipate deviations and distinguish planned from unplanned work.

Product teams do not need to publish plans or adopt an academic template. The transferable mechanism is smaller: preserve what was specified before outcome data could influence the specification.

An edit made after someone inspected the result is not automatically wrong. It is a new fact about the analysis and should be labelled as such.

Keep eligibility, assignment, delivery and exposure separate

These states answer different questions. Combining them makes missing users invisible.

Eligibility means the unit qualified under a named rule and version. Record when the rule was evaluated, the data it used and exclusions known before treatment could affect behaviour.

Assignment means the randomisation unit received a variant allocation. Store the unit type, assignment timestamp, expected ratio, bucketing version and assigned variant.

Delivery means a particular build or configuration returned the assigned experience. A successful assignment does not prove the product delivered it.

Exposure means the unit had the opportunity defined in the plan to encounter the intervention. A component render, processed request and sent message are different exposure claims.

Preserve the assigned population even when analysis also uses an exposed population. If treatment can affect whether exposure is recorded, filtering to exposed units can make the groups incomparable.

The tracker should show which population each analysis used and why. It should never let “participants” mean assigned users in one result and exposed users in another.

Also record collisions: concurrent experiments, cross-device reassignment, shared accounts, cache behaviour, opt-outs and product versions that cannot deliver the treatment.

These are not footnotes. They define what comparison was actually possible.

Treat telemetry health as evidence about the evidence

Primary outcomes and guardrails should reference controlled metric definitions, not copied labels. The record needs the definition version, query or computation version, effective dates and known coverage gaps.

Guardrails have a different job from the primary outcome. They define changes the team is unwilling to ignore while evaluating the intended benefit.

The tracker preserves which guardrails were specified before the run, their owners, review conditions and disposition. Adding one after seeing a movement must be visible as a post-start addition.

Before interpreting an effect, inspect at least:

• eligible, assigned, delivered and exposed counts by variant;
• expected versus observed assignment ratios;
• missing, duplicate and late records;
• join rates across assignment, exposure and outcome data;
• coverage by client, service and release version;
• metric-computation failures and schema breaks;
• telemetry changes during the run;
• parity of observation paths across variants.

A sample-ratio mismatch, or SRM, occurs when observed variant counts depart from the configured ratio beyond what the chosen check expects from chance.

Fabijan and co-authors derived an SRM taxonomy from work across four software companies and more than 25 products. They found possible causes in assignment, execution, log processing and analysis.

That experience base does not estimate how common SRM is in every product. Its practical value is diagnostic: SRM is a symptom, not a root-cause label.

Do not repair it by normalising counts or removing an inconvenient segment. Record the affected interval, investigation, identified mechanism and consequence for each analysis.

Telemetry volume can look balanced while observation is still biased. Gupchup and co-authors modelled telemetry loss and applied their framework in Microsoft applications operating at large scale.

Their paper shows that the effect of loss depends on its mechanism. Under some conditions it reduces power; under others it can bias estimates.

The result is not a universal correction formula. It is a reason to preserve loss diagnostics and avoid treating “missing at random” as an invisible default.

Record deviations without rewriting the past

Real runs change. A rollout pauses, a bug fix alters delivery, an eligibility source fails, or an incident forces an early stop.

Use an append-only deviation log. For each deviation, record:

• time and affected run;
• what differed from the frozen plan;
• who initiated and approved it;
• operational or evidential reason;
• data inspected before the change;
• populations, metrics and time ranges affected;
• whether the current run remains interpretable;
• resulting amendment, new run, exclusion or stop.

Separate operational monitoring from inferential stopping.

A team may pause immediately for harm, security, privacy or service failure. That decision protects users; it does not turn the partial effect estimate into decision-ready causal evidence.

If a fixed-horizon analysis was planned, repeatedly inspecting ordinary fixed-horizon p-values and stopping when one crosses a threshold changes the procedure.

Johari, Koomen, Pekelis and Walsh developed always-valid inference for sequential monitoring under stated statistical assumptions. Their work does not make every dashboard safe to monitor continuously.

The tracker therefore records the planned inference family, actual review times, stop trigger, stop authority and data visible at the decision. A method changed after inspection should not inherit the original label.

Make analysis lineage executable

Copying a result into a document breaks the chain at its most important point.

For each analysis, preserve these items or reference immutable versions:

• plan, run and metric-definition versions;
• assignment, delivery and exposure inputs;
• source tables or dataset snapshot and extraction time;
• identity, joins, windows, exclusions and corrections;
• query, notebook, package and code revisions;
• statistical method and configured parameters;
• prespecified and exploratory outputs kept distinct;
• analyst or service responsible;
• execution time, review status and superseded analyses.

Another qualified person should be able to rerun the analysis or explain exactly why it cannot be rerun.

The W3C PROV-O Recommendation models provenance through entities, activities and agents, with relations for use, generation, derivation, attribution and revision.

PROV-O is an interoperability ontology for web provenance, not an experimentation standard. A product team does not need RDF to borrow its useful discipline.

The plan, dataset, metric contract and code are entities. The analysis job is an activity. The analyst or software service is an agent. The result is derived from named inputs and can be superseded by a revision.

That graph is stronger than a folder of screenshots because it preserves responsibility and transformation, not only storage.

Make the decision a separate object

An experiment result and a product decision are not the same record.

The analysis may estimate an effect while the decision also considers guardrails, operational burden, strategic fit, reversibility, legal duties and evidence outside the experiment.

Create a decision entry with:

• decision, date, owner and authority;
• result and analysis versions reviewed;
• telemetry-health and deviation status;
• evidence limits and unresolved disagreements;
• chosen action: adopt, reject, revise, rerun, hold or make no change;
• rollout conditions and accountable owners;
• expected follow-up evidence;
• trigger for reopening the decision.

“Inconclusive” can describe evidence. It does not describe what the team chose to do next.

Link follow-up observations to the decision rather than editing the old result. Later guardrail movement, a failed rollout condition or a changed metric definition should create new evidence and, if needed, a new decision.

The approved data-informed decision guide covers how to weigh evidence and alternatives. Experiment tracking preserves which evidence that decision actually used.

Use the catalogue before and between runs

A reconstructable record should also stop the team from rediscovering the same failure.

Before approving a new experiment, search by assumption, population, product state, metric contract, intervention and failure mode.

Review the catalogue periodically for patterns worth investigating:

• assumptions that fail across several decision contexts;
• effects that reverse for a named segment;
• recurring assignment, exposure or telemetry defects;
• methods that repeatedly produce decision-weak evidence;
• follow-up conditions that were promised but never observed.

Do not pool wins or average effects merely because experiments share a label. Different populations, treatments, metric versions and analysis designs may make the results incomparable.

Treat cross-run patterns as candidates for a stronger study, not as a meta-result created by a dashboard. The catalogue earns its keep when it changes the next question.

Design privacy into the record

Assignment and exposure logs can connect identity, time, product behaviour and treatment. A reproducible experiment is not permission to retain every row indefinitely.

For each collected field, record its purpose, access, retention, deletion handling and whether a less identifying representation can support the analysis.

Keep the ability to join a person to raw exposure data separate from the durable experiment catalogue where possible. Restricted identifiers are still data with consequences; renaming them does not make them anonymous.

Record how consent, objection, opt-out or deletion changes assignment, exposure and outcome coverage. The analysis must not present the remaining population as complete when collection conditions exclude part of it.

W3C’s 2025 Privacy Principles is a Statement prepared for the web, not legislation or a product-experiment compliance standard. It addresses data minimisation, specified purposes, secondary use and user controls.

Use it to challenge collection architecture, then involve privacy, security and legal specialists for the actual product, people and jurisdictions.

A fictional experiment record

The following case is fictional and has no claimed outcome.

A B2B project-management product tests an in-product reminder for project owners whose review is overdue.

The permanent experiment ID holds the question. The first run records workspace as the randomisation unit, the workflow states eligible at assignment, the allocation configuration and the two release builds.

Delivery means the assigned component configuration reached the client. Exposure requires the owner to open the review page and the reminder to render. The record keeps assignment and exposure populations separate.

The primary outcome references a versioned review-completion definition. Guardrails reference reminder dismissal, accessibility errors and support contacts. No thresholds are invented in the catalogue.

Telemetry-health checks cover expected assignment ratio, assignment-to-delivery joins, render coverage by client version and outcome-event parity across variants.

The plan states that changing eligible workflow states creates a new run. A rendering defect would enter the deviation log with its interval and affected versions before any analysis is accepted.

The analysis object points to the frozen inputs, code revision and method. The decision object remains empty until an authorised review records an action and follow-up conditions.

The catalogue stores bounded metadata and access-controlled references. It does not duplicate raw user-level exposure data for convenience.

Nothing in this example demonstrates effectiveness. It shows what must remain connected for any later claim to be challenged.

Preserve the chain, not just the conclusion

A trustworthy experiment record lets a practitioner move in both directions.

From the plan, they can follow the run through eligibility, assignment, delivery, exposure, telemetry, analysis and decision.

From a product decision, they can trace back to the exact evidence, versions, deviations and unresolved limits that supported it.

That is the value of experiment tracking: not institutional memory as folklore, but a reconstructable chain of responsibility and evidence.

Sources

• OSF Support, “Welcome to Registrations & Preregistrations” — official OSF workflow guidance for research registration. The article borrows the time-stamped, read-only plan mechanism; it does not equate an internal product record with public academic preregistration.
• W3C, “PROV-O: The PROV Ontology” — W3C Recommendation for interoperable provenance information. Its entity, activity, agent, derivation and revision concepts inform lineage here; it is not an experiment-tracking specification.
• Fabijan et al., “Diagnosing Sample Ratio Mismatch in Online Controlled Experiments” — taxonomy derived from experience across four software companies and more than 25 products. It does not establish universal SRM prevalence or one root cause.
• Gupchup et al., “Trustworthy Experimentation Under Telemetry Loss” — theoretical treatment plus Microsoft application cases at scale. Its conclusions depend on loss mechanisms and do not supply a generic repair for missing product data.
• Johari, Koomen, Pekelis and Walsh, “Always Valid Inference: Continuous Monitoring of A/B Tests” — sequential-inference methods under specified statistical assumptions. It supports recording the actual stopping method, not switching methods after observing results.
• W3C, “Privacy Principles” — 2025 W3C Statement for the web. It is a consensus principles document, not legislation, a conformance standard or jurisdiction-specific experiment guidance.

Read next

• Conversion Tracking: Build a Trustworthy Measurement Contract
• Engagement Analytics: Measure Evidence of Value, Not Motion
• Making Smart Decisions: Data-Informed Product Management