User Feedback in Product Decisions: Represent, Do Not Count

A product council reviews forty requests for configurable approvals and four reports that the current workflow can expose confidential records.

The larger pile wins. The team calls the decision customer-led.

That is not a feedback-informed decision. It is a vote in which the ballot box, electorate, and cost of being ignored were never defined.

Including user feedback means representing relevant experiences inside a specific decision. It does not mean obeying every request, averaging incompatible preferences, or letting the loudest channel set the roadmap.

The hard work is editorial and political: decide whose experience matters to this choice, show how each account was selected and interpreted, preserve disagreement, and make the final judgement visible.

Start with the decision, not the feedback pile

“What are users asking for?” is usually too broad to govern a product choice.

Name the decision before retrieving evidence:

Decision: the choice that is still open
Population: people who use, buy, operate, support, or are affected by it
Consequence: what each group can gain, lose, or be required to do
Alternatives: materially different responses still available
Boundary: constraints that are fixed, negotiable, or only assumed
Authority: who recommends, challenges, and decides
Decision date: when evidence must be usable
Reopening condition: what would justify another decision

This contract changes which feedback is relevant.

A packaging decision may need evidence from buyers, administrators, finance, and users who never see the invoice.

An access-control decision may also affect security reviewers, support agents, data subjects, and people unable to use the current recovery route.

Starting with the inbox instead would let the available channel define the population by accident.

Build a representation map

User feedback is never simply “the voice of the customer.” It is a set of accounts produced through different opportunities to speak.

For the decision, map the groups whose situations could change:

• direct users performing the work;
• buyers, administrators, or approvers who create conditions for that work;
• people who attempted the task and abandoned it;
• people excluded by access, language, device, price, policy, or eligibility;
• service, support, operations, and implementation staff who see failure downstream;
• people affected by the product without being its customer or active user.

Then attach each feedback source to the groups it can and cannot reveal.

Support tickets can expose recoverable incidents among people able to reach support. Sales notes expose commercial conversations after a salesperson has selected and summarised them.

An in-product prompt reaches people who arrived at the prompt in the measured version. A cancellation survey excludes people who leave without completing it.

The GOV.UK Service Manual recommends defining target groups and recruitment criteria around the research questions, then including likely users with different access needs and circumstances.

That is public-service research guidance, not a sampling rule for every product. The useful discipline is to design inclusion rather than equate availability with relevance.

Mark every important group as represented, weakly represented, or missing. Do not fill an empty cell with an invented persona or one colleague’s intuition.

Missing representation is itself decision information. It may justify targeted research, a safer alternative, a limited rollout, or an explicit acceptance of uncertainty.

Record how the channel selected the speaker

A feedback source is a selection mechanism as well as a container.

Pagano and Maalej analysed 1,126,453 reviews across 1,100 Apple App Store applications, split evenly between free and paid, in a 2013 requirements-engineering study.

Because the App Store reset an application’s reviews with each release, the dataset contained only reviews posted after each application’s latest release. Of all reviews, 518,041 (45.99%) specified the reviewed version.

Using reviewer names as identifiers, the authors counted 918,433 reviewers. Of those, 826,874 (90.03%) posted once, while 1,183 (0.13%) posted more than five reviews.

Those identifiers included anonymous usernames, so the distribution describes recorded reviewer names rather than verified individuals.

Those observations describe that historical App Store dataset. They do not establish the distribution of opinion in current app stores or in a particular product.

They do show why a large channel can still have a narrow evidential boundary. Volume does not identify the silent population, reconstruct exposure, or make the underlying product state comparable.

For every source used in a decision, retain:

• who could encounter the channel;
• who could realistically respond;
• what event prompted the response;
• the product, policy, account, and time state;
• whether an intermediary translated the account;
• which relevant groups could not appear;
• the denominator, if a rate or prevalence claim is made.

“Twenty-seven customers requested this” is a count. “Twenty-seven of 140 administrators contacted after a failed approval reported the same permission boundary” is a bounded observation.

Neither statement decides what to build.

Type the claim before weighing it

Feedback records often mix several claim types in one sentence. Separate them because each can support a different inference.

Event report

What the person says happened in a particular context: “The approver opened the link and saw records from another department.”

It may need reproduction or corroboration, but it has a concrete event boundary.

Preference

What the person says they would choose: “I prefer a weekly digest.”

Preferences can inform design, especially when trade-offs are explicit. They do not reliably predict behaviour under different cost, effort, or social conditions.

Proposed solution

What someone wants the product to do: “Add approval templates.”

Preserve the request, then reconstruct the task, failure, and consequence before treating it as a response to the right problem.

Obligation or constraint

What a contract, regulation, policy, workflow, or dependency requires: “Each approval must retain an auditable decision record.”

Verify the source and authority. A user’s interpretation may reveal a real constraint, a local practice, or a misunderstanding.

Prediction

What someone expects to do or what they believe will happen: “Our team would upgrade for this.”

Treat it as a prediction until the relevant choice, price, and conditions are real enough to test.

Observed consequence

What followed the event: delay, workaround, financial loss, exposure, abandonment, support work, or another measurable effect.

Record whose consequence it was and how it was observed. Do not silently turn inconvenience into harm or a reported benefit into a causal result.

Typing the claim prevents a vivid quote, a contractual obligation, and a speculative feature request from becoming three identical votes.

Separate frequency from consequence

Frequency matters when the population, opportunity, time window, and event definition are defensible. It is not a universal measure of importance.

Griffin and Hauser tested whether mention frequency could stand in for measured importance within a 1993 Voice of the Customer study of portable food-carrying devices.

In that setting, the needs measured as important were no more likely to be mentioned than needs in general. The authors concluded that mention frequency was not a good surrogate for importance.

This is one product-research setting inside a wider Quality Function Deployment study. It does not prove that counts are useless across products.

It does invalidate the shortcut “most mentioned means most important.”

Judge frequency beside other properties:

• reach: how much of the relevant population encounters the condition;
• consequence: what happens when it occurs and who bears it;
• recoverability: whether the person can detect, reverse, or escape it;
• inequality: whether the burden concentrates on a group poorly represented in the channel;
• strategic relevance: whether it changes an assumption behind the current choice;
• obligation: whether law, contract, safety, rights, or service continuity changes the decision threshold.

A rare permissions leak and a common colour preference should not enter the same popularity ranking.

This is not permission to label a favoured request “strategic” and bypass scrutiny. State the decision property that changes its weight and who has authority to accept the trade-off.

Construct an evidence panel, not a confidence score

Bring feedback into the decision as a panel of inspectable evidence.

For each materially different alternative, write:

Strongest supporting feedback
Strongest contradicting feedback
Relevant group not represented
Claim types being relied on
Behavioural or operational evidence that bears on the claim
Potential high-consequence case hidden by aggregation
Interpretations still competing
Evidence that would change the recommendation

Do not average interviews, usage events, tickets, commercial notes, and survey responses into one confidence number.

They observe different things through different instruments. Behavioural data can show recorded actions among instrumented units.

Feedback can reveal goals, explanations, constraints, language, and consequences the instrumentation cannot see. Operational records can reveal failure and recovery work.

Their agreement can strengthen a bounded claim. Their disagreement can be more useful: it may expose a segment difference, an instrumentation gap, a workaround, or a mistaken interpretation.

The GOV.UK guide to analysing research separates observation, grouping, interpretation, and action. Its exact workshop format belongs to government-service research.

The transferable discipline is keeping a route from a proposed action back through the finding to what was actually observed.

Preserve minority and contradictory accounts

Synthesis becomes dangerous when it turns disagreement into a smooth theme.

A minority account may be:

• noise or a misunderstanding;
• evidence from a different product state;
• a weak signal of a severe failure;
• the only visible account from an excluded group;
• a counterexample that breaks the team’s explanation;
• a legitimate preference conflict with no design that satisfies everyone.

Do not automatically elevate or discard it. Record why it differs and which decision property makes the difference relevant.

Use contradiction labels that force investigation: different actor, different task, different state, different consequence, different interpretation, or unresolved.

Invite someone to argue the strongest case against the emerging recommendation. Their job is not generic devil’s advocacy.

They should point to excluded evidence, unsupported transformations, hidden costs, and groups who bear consequences without decision power.

Make judgement visible

Feedback does not decide. People with authority decide, under uncertainty, with consequences for other people.

The decision record should state:

1. the choice and alternatives considered;
2. the represented and missing groups;
3. the feedback claims used and their selection limits;
4. corroborating and contradicting evidence;
5. the consequence and obligation analysis;
6. the recommendation, dissent, and final authority;
7. what was not inferred from the evidence;
8. the response, verification, and reopening conditions.

This record protects against two opposite failures.

The first is customer theatre: displaying quotes while the decision follows an unstated internal preference. The second is feedback literalism: treating a request as a delegated product decision.

A team may responsibly decide against a requested solution. It should still be able to show that the underlying situation entered the choice and was not erased by channel volume or organisational convenience.

How to Prioritise covers the adjacent trade-off once evidence, obligations, capacity, and strategic fit must be compared across options.

A fictional decision with an inconvenient minority

Consider a fictional workflow product deciding whether to build reusable approval templates.

Forty requests come from administrators who repeatedly configure similar approvals. Four support reports concern reviewers seeing confidential fields outside their role.

The representation map shows that requesters are mostly administrators from large accounts. Reviewers and data subjects have no direct feedback channel.

The forty records are proposed solutions attached to a recurring setup task. The four are event reports with a potentially high consequence, but two concern an older permission model and none has yet been reproduced.

Usage data shows repeated configuration but cannot reveal whether templates would be safely reusable across departments. Support evidence reconstructs one permission boundary; security review identifies another plausible exposure path.

The team does not vote forty to four. It compares alternatives: ship templates broadly, limit them to one role model, first repair the permission boundary, or test a constrained template with selected accounts.

The decision owner chooses the constrained test only after defining which fields and roles may be copied. The unresolved reports receive a separate investigation route.

No outcome is claimed. The example shows why inclusion changes a decision even when the most popular request remains relevant.

Hand the decision back to the loop

After the choice, tell contributors what the team understood, what it decided, and what remains uncertain without inventing a roadmap promise.

Define how the original situations will be checked after any change. A shipped feature does not prove that a workflow, exclusion, or harm improved.

User Feedback Loops covers ownership, disposition, response, verification, expiry, and reopening after a signal enters the system.

The boundary matters. This article governs representation inside one product decision. A feedback loop governs the path through which signals are received, handled, answered, and checked.

The minimum representation record

For every consequential product decision that claims to include user feedback, retain:

• the decision, alternatives, population, consequence, and authority;
• the representation map and known missing groups;
• the selection mechanism and state of each feedback source;
• separated event, preference, solution, obligation, prediction, and consequence claims;
• frequency with a defensible denominator where one exists;
• contradictory and high-consequence cases;
• the evidence panel and limits on inference;
• the judgement, dissent, response, verification, and reopening condition.

User feedback deserves more than a vote count or a quotation on a slide.

It deserves a visible place in the reasoning—and an honest account of whose experience still did not make it into the room.

Sources

• Griffin and Hauser: The Voice of the Customer (peer-reviewed 1993 Marketing Science study in a Quality Function Deployment context; the mention-frequency result used a nine-point importance measure in one portable food-carrying-device study)
• Pagano and Maalej: User Feedback in the AppStore—An Empirical Study (peer-reviewed exploratory paper presented at the 2013 IEEE International Requirements Engineering Conference; historical Apple App Store dataset using reviewer names as identifiers)
• GOV.UK Service Manual: Finding participants for user research (government-service guidance published in March 2016 and updated in April 2020; covers target groups, recruitment criteria, inclusion, access needs, and recruitment bias)
• GOV.UK Service Manual: Analyse a research session (government-service guidance published and last updated in May 2016; separates observation, grouping, interpretation, and action)

Read next

• User Feedback Loops: Close the Decision, Not the Ticket

• Unsolicited User Feedback: From Raw Request to Product Decision

• How to Prioritise: Make Trade-offs, Not Just Rankings

• When Product Research and Analytics Disagree