Analytics Infrastructure: Reconstruct the Evidence Path

A product number can survive a broken path. It can look plausible in a dashboard, match last week’s trend, and still describe the wrong people, the wrong moment, or an old version of the product.

Analytics infrastructure is the controlled, reproducible path from observed behaviour to an answer. Its job is to keep a claim inspectable while products and pipelines change.

That path includes product state, event meaning, identity, time, eligibility, consent, source and transformation versions, quality checks, analysis code, and owners.

This is narrower than a tool stack or a universal architecture. It is also separate from metric governance, dashboard design, and the broader capabilities of a data platform.

Begin with one claim boundary

“Capture more data” is not a starting point. Start with one decision and the smallest claim that could inform it.

A claim boundary states what the answer may describe and what it may not imply. It stops a query from quietly expanding beyond the evidence available.

Use this review block before specifying events:

Decision: What choice could change because of this analysis?
Unit: What entity is counted: person, account, device, order, or session?
Population: Who is eligible, and at what point in time?
Behaviour: What observable action or state supports the claim?
Time basis: Which event-time window and reporting cut-off apply?
Comparison: What is being compared, if anything?
Claim: What may the result reasonably say?
Exclusions: Which interpretations would exceed the evidence?
Uncertainty: Which missing, late, or ambiguous records are tolerable?
Owner: Who approves the answer for this decision?

Suppose the decision is whether to keep a revised approval reminder. “Eligible accounts completed approval after seeing the reminder” is narrower than “the reminder improved activation”.

The second statement introduces causality and a broader outcome. The evidence path may support the first while being incapable of supporting the second.

The meaning and governance of a metric belong in Building a Product Metrics Practice. Here, the concern is whether the observations behind an answer remain reconstructable.

Trace the full evidence path

An event name is only one stop. Draw the route before debating storage or vendors.

For one claim, trace:

1. the product question and decision owner;
2. the observable behaviour in a specific product version;
3. the captured observation and its contract version;
4. identity, eligibility, and consent state at event time;
5. ingestion, deduplication, and late-arrival handling;
6. validated source data and its quality state;
7. transformations, joins, and cohort rules;
8. the analytical dataset or snapshot;
9. query or notebook version, parameters, and reviewer;
10. the answer, its trust state, and the decision record.

This is an evidence map, not a prescribed system diagram. Its implementation can live in a few jobs or across several services.

The broader capability and operating model belong in Understanding Data Platforms for Non-Technical PMs.

Give every observation a contract

“Button clicked” is not a semantic definition. It does not say which button state, which user state, whether the click was accepted, or whether retries create another record.

For each decision-critical event or state change, record:

• Semantic trigger: the product condition that creates the observation, including failures and cancellations.
• Unit: the entity represented by one record, such as a person, account, device, order, or session.
• Identity state: anonymous, authenticated, merged, delegated, shared, or otherwise relevant.
• Event time: when the behaviour happened in the product, including timezone and clock assumptions.
• Processing time: when the observation reached each controlled stage of the pipeline.
• Contract version: the schema and meaning active when the event was emitted.
• Required properties: fields needed to interpret or join the observation, with type and allowed values.
• Consent or eligibility state: why this observation may be collected and included in the intended analysis.
• Retry and deduplication expectation: the stable key and rule that distinguish retry from repeated behaviour.
• Owner: the person or team responsible for meaning, implementation, and operational response.
• Verification evidence: the test, fixture, log, or inspected trace proving that the product emits the intended record.

The verification evidence matters. A contract checked only against sample rows proves that rows exist. It does not prove that the right product condition created them.

Contract changes need an effective date and compatibility decision. Reusing the same event name for a new meaning erases the boundary between product versions.

Identity and time are analytical inputs

Event time and processing time answer different questions. Event time places behaviour in the product journey. Processing time tells the pipeline when the record became available.

A reporting cut-off must state which one controls inclusion. Otherwise late records can silently rewrite a cohort, while a retry can masquerade as new behaviour.

Define the response to four ordinary conditions:

• Late: include in the original event-time period, a later processing period, or a stated restatement?
• Duplicate: which stable identifier and window make two records the same observation?
• Missing: when is absence treated as “did not happen”, “not yet received”, or “cannot be known”?
• Out of order: which event or product state resolves an impossible sequence?

An anonymous device may later map to a person. People may share an account. Accounts may merge, split, change plan, or lose eligibility.

Keep the raw identity state and the rule used for analytical resolution. Do not overwrite history with the latest account profile and call it the state that existed when the behaviour occurred.

Cohort membership should be reproducible “as of” a defined time. If the membership rule changes, version the rule and record whether historical cohorts will be restated.

Use the canonical funnel guide to interpret sequences.

The cohort and retention guides own their respective methods.

Version the path, not only the output

The reconstruction record should identify:

• source tables, partitions, snapshots, or extraction queries;
• event and source schema versions;
• transformation code commit and configuration;
• identity, eligibility, cohort, and exclusion rule versions;
• parameters, time windows, timezone, and cut-off;
• reference data versions, such as plan or market mappings;
• backfills, corrections, and known exceptions;
• the analytical query or notebook commit;
• output location, creation time, owner, and review state.

A backfill is a change to evidence, not routine housekeeping. Record its reason, affected observations, time range, code version, quality checks, and downstream answers that may need restatement.

Preserve the original answer when a decision has already used it. Publish the revised answer beside it, with an explanation of what changed and whether the decision should be revisited.

Do not pretend that today’s cleaner dataset was the dataset available at the time.

Use lineage to answer an incident

Useful lineage connects an answer to concrete inputs, transformations, and owners. A diagram that ends at generic boxes is decoration.

The W3C PROV-O Recommendation offers a durable vocabulary: entities are things, activities act on things over time, and agents bear responsibility.

That model can frame a lean provenance record:

• source observations and reference snapshots are entities;
• ingestion, deduplication, transformation, and analysis are activities;
• services, teams, or named owners are agents;
• relations record what was used, generated, derived, attributed, and associated.

PROV-O is an OWL2 ontology and reference model for interoperable provenance. A team can use its vocabulary as a review lens without adopting RDF, and the Recommendation does not show that any specific lineage tool improves decisions.

Treat quality as fitness for this claim

“The data is clean” is not a release criterion. Quality depends on the claim and decision.

The UK Government Data Quality Framework defines quality through fitness for intended use. It describes six dimensions: completeness, uniqueness, consistency, timeliness, validity, and accuracy.

Turn those dimensions into claim-specific gates:

• Completeness: are the required observations and fields present for the eligible population?
• Uniqueness: are retries or duplicated deliveries counted once under the agreed rule?
• Consistency: do identity, state, and totals agree across the sources this claim combines?
• Timeliness: did enough data arrive before the cut-off for this decision?
• Validity: do records satisfy the contract’s types, ranges, and allowed values?
• Accuracy: does verification evidence show that records correspond to the product behaviour?

The framework’s guidance notes that timeliness can trade off against accuracy or the amount of data available. That is why “fresh” cannot be a universal quality target.

Declare a trust state on every promoted dataset and answer. A workable vocabulary is verified, provisional, quarantined, and retired.

Verified means the stated gates passed for the named use. Provisional means a known limitation remains visible. Quarantined means the evidence must not support the decision. Retired means the path is preserved only for history.

Quarantine should reach the consumer through a banner, metadata field, or blocked refresh, not remain inside a pipeline alert.

Schelter and colleagues show one production-oriented way to express constraints declaratively and evaluate them through aggregated checks at scale.

The paper supports automating defined checks. It does not establish a universal architecture, automatic repair, or business impact.

Make the answer reproducible

The UK Government Analysis Function defines reproducibility as obtaining the same result with the same methods and the same data.

Its Reproducible Analytical Pipelines strategy recommends version control, documented code, quality assurance, peer review, and limiting manual steps. It is public-analysis guidance, not a study of product teams.

For a product answer, package:

• an immutable input snapshot or an extraction query with a stable cut-off;
• all source, contract, transformation, and cohort versions;
• analysis code and dependency versions;
• parameters and environment assumptions;
• documented manual steps;
• output checksum or equivalent identity;
• reviewer, review date, trust state, and known limitations.

Reproducibility does not make an inference valid. It lets a reviewer recreate the answer and inspect where assumptions entered.

Control migration before it changes history

Instrumentation and pipeline migrations alter evidence. Treat them as product changes with consumers and compatibility risks.

Before release:

1. name the old and new contract or transformation versions;
2. define which populations and product versions each covers;
3. run both paths over an overlap where feasible;
4. compare contract-level counts, identity states, timing, and representative traces;
5. decide how late records cross the boundary;
6. document whether historical data will be backfilled;
7. notify owners of affected datasets, analyses, and decision surfaces;
8. set the retirement condition for the old path.

Do not delete the old definition when the code stops emitting it. Retain its active dates, known limitations, downstream consumers, and location of historical evidence.

Retirement is complete when no active consumer depends on the path, open incidents are resolved, required history remains reproducible, and the replacement has a named owner.

Diagnose incidents from the claim backwards

When an answer looks wrong, begin at the claim boundary. Changing queries until the number looks familiar destroys evidence.

Check in this order:

1. Did the decision, unit, population, or time basis change?
2. Is the answer using the intended dataset and analysis version?
3. Did a cohort, identity, consent, or eligibility rule change?
4. Was there a transformation release, backfill, or restatement?
5. Did late, duplicate, missing, or out-of-order data cross a threshold?
6. Did the source contract or product behaviour change?
7. Does trace evidence connect representative records to the product condition?

Record the fault, affected claims, time range, trust-state change, containment, correction, and restatement decision. Close the incident when consumers know which answers remain usable.

Fictional case: the approval reminder

The following case is fictional. It illustrates the review protocol and contains no measured result.

A B2B product team is deciding whether to keep a revised approval reminder. Its bounded claim concerns eligible accounts that saw the reminder and later completed approval within the defined event-time window.

The view event contract records reminder version, account identity state, eligibility state, consent state, event time, delivery identifier, product release, and verification trace.

The completion event records the approved object, responsible account, event time, and stable operation identifier. Retries are deduplicated by that operation identifier.

During review, the team finds that an account-merge migration can attach earlier anonymous activity to the current account. Some events also arrive after the reporting cut-off.

The team does not produce a cleaner-looking answer by overwriting identity or dropping late rows. It versions the merge rule, states the late-arrival policy, and marks the answer provisional.

The revised pipeline runs beside the old path over an overlap. Representative traces are checked against product logs. A backfill plan names the affected period and the analyses that would need restatement.

Until those checks pass, the team cannot use this path to decide whether the reminder improved anything. The infrastructure has done its job by exposing the limit before the claim hardens into fact.

Run an evidence-path review

Use this protocol for a decision-critical analysis, migration, or incident. Run it when evidence changes, not by ceremonial calendar.

Ask:

1. Is there one explicit decision and claim boundary?
2. Can each observation be tied to a semantic trigger and product version?
3. Are unit, identity, eligibility, consent, event time, and processing time preserved?
4. Are retry, deduplication, missing-data, and late-arrival rules explicit?
5. Can the answer identify every source and transformation version?
6. Are backfills and restatements visible rather than overwritten?
7. Do quality gates reflect fitness for this claim?
8. Is the trust state visible where the answer is consumed?
9. Can another reviewer reproduce the answer from preserved inputs and code?
10. Is there an owner for migration, incident response, and retirement?

A dashboard is one possible consumer of this path. Designing the recurring decision surface is covered in Data-Driven KPI Dashboards for PMs.

The infrastructure review ends when the team can show what the answer means, how it was produced, which limitations remain, and who is responsible for changing its trust state.

Sources

• The Government Data Quality Framework, published 3 December 2020, is official UK guidance. It defines fitness for purpose, describes six core dimensions, and says the list is non-prescriptive. It is not an empirical product-analytics study.
• Framework guidance explains how to assess the dimensions, including trade-offs between timeliness, accuracy, and the amount of data available. It reports guidance, not product outcomes.
• PROV-O is a W3C Recommendation for representing provenance through entities, activities, agents, and their relations. This article uses its vocabulary as a review lens; it does not claim PROV-O conformance, require RDF, or treat the Recommendation as evidence that a lineage tool is effective.
• The Reproducible Analytical Pipelines Strategy, published 20 June 2022, is UK government strategy for public analysis. Its practices are guidance, not causal evidence about product decisions.
• Schelter et al., PVLDB 2018 present declarative checks implemented as Spark aggregations and experimentally evaluate scalability, incremental computation, and constraint suggestions.
• Their samples include about 120 million internal product-catalogue records, plus public Reddit and Twitter data. They test scalability and suggestions and demonstrate an anomaly-detection example, not decision quality or business impact.
• The paper reports useful checks alongside suggestion errors and leaves concerns such as streaming for future work. Its architecture and findings should not be generalised to every analytics system.

Read next

• Building a Product Metrics Practice
• Understanding Data Platforms for Non-Technical PMs
• Data-Driven KPI Dashboards for PMs
• ML Deployment: The Product Manager’s Release Contract