Personalisation Engines: Define the Decision Before the Model

Every personalised experience makes a decision.

It chooses which item, message, route, offer, or action one person receives instead of the available alternatives. The decision can come from a rule, a segment, a ranking model, or a human editor.

Calling the result personalised does not make the decision useful. It can narrow a person’s options, repeat what is already popular, expose sensitive inferences, or optimise a click that leaves the underlying experience worse.

The work begins before model selection. Define the decision policy: who is eligible, what may be selected, which evidence is permitted, what the system should optimise, and what happens when it does not know enough.

Define the decision and the user benefit

“Personalise the homepage” is a surface, not a product decision.

Name the choice at the smallest useful level. The product might rank learning modules after onboarding, select a setup path from a stated role, or suggest a document related to the one currently open.

Then name the benefit in the user’s terms. Does the policy reduce search effort, help someone find an unfamiliar but relevant option, protect them from noise, or sequence work around a current goal?

If the only answer is more engagement, the system has no standard for distinguishing useful attention from compulsive or accidental interaction.

Write a policy statement:

For [eligible person and context], choose [item or action] from [candidate set] using [permitted evidence] to improve [user outcome], while protecting [guardrails].

The statement exposes decisions that “recommend relevant content” conceals. It also gives design, data, engineering, privacy, and operations a shared object to challenge.

Ask whether context, segments, or rules are enough

One-to-one ranking is not the starting point for every variation.

Context may carry most of the useful information. A person reading a contract template may benefit from guidance attached to that template, regardless of their long-term behaviour.

A small set of segments may represent genuinely different jobs. An administrator and a frontline user can need different setup paths because their responsibilities differ, not because a model detected a correlation.

Rules are appropriate when the logic should be stable and intelligible. Eligibility, contractual entitlement, quiet hours, safety exclusions, and explicit preferences should not be rediscovered from behaviour.

Use a more adaptive policy when there are many plausible candidates, repeated decisions, enough relevant evidence, and meaningful variation that simpler approaches cannot express economically.

Google’s Rules of Machine Learning recommends building a solid pipeline, sensible objective, and simple baseline before increasing model complexity.

That is more than engineering advice. A baseline tells the product team whether adaptation earns its operational and user cost.

Write the data and eligibility contract

Personalisation turns recorded behaviour and attributes into product treatment. That deserves an explicit contract.

For every input, record:

• where it comes from and who owns it;
• what it means in the product, including known ambiguity;
• whether it is fresh enough for this decision;
• which people and contexts it covers or misses;
• whether consent, purpose, retention, or access limits apply;
• what the policy does when the value is absent or disputed.

Do not treat a click as a preference without examining the conditions that produced it. The item may have been placed first, shown repeatedly, or chosen because every alternative was worse.

Separate eligibility from ranking. A model may score plausible content, but a policy layer should remove items that are unavailable, unsafe, already completed, legally restricted, or contrary to an explicit choice.

The contract also needs an identity model. Decide what happens across devices, shared accounts, changing roles, anonymous sessions, and deleted history.

Weak identity resolution can make a sophisticated ranker confidently personalise for the wrong person.

Understand candidates, scoring, filtering, and fallback

A recommendation is the result of a system, not one score.

Google’s recommendation-systems overview describes candidate generation, scoring, and re-ranking as a common large-scale architecture.

The exact architecture belongs to the technical team, but product decisions shape every stage.

Candidate generation

The candidate set defines what the user could possibly receive. It might come from a catalogue, current context, similar items, editorial selection, business rules, or more than one source.

If the right item never enters the set, better ranking cannot recover it.

Scoring

Scoring estimates which eligible candidates best serve the objective. The features and labels embody choices about relevance and value.

A score trained on clicks predicts behaviour under the old interface and policy. It does not automatically represent satisfaction, understanding, or long-term benefit.

Re-ranking and filtering

The final policy may enforce diversity, freshness, fairness, contractual obligations, safety, or a limit on repeated content. These are product requirements, not cleanup after “the real model.”

Fallback

When data is missing, the service is unavailable, or confidence is insufficient, the experience needs a useful default.

The fallback might use current context, a transparent popularity measure, an editorial list, a simple segment, or the unpersonalised experience.

Designing the fallback early prevents the model from becoming the only path through the product.

Choose an objective and guardrails

Optimisation makes a proxy more likely, not the full product outcome.

Click probability is observable and attributable. It may still reward sensational content, repeated exposure, or short-term curiosity.

Completion, qualified discovery, retained use, or successful task progression may sit closer to value. They also arrive later and can be harder to attribute.

Choose an objective the system can learn from, then surround it with product measures that decide whether a launch is responsible.

Useful guardrails can include:

• hides, dismissals, and explicit negative feedback;
• repeated exposure and catalogue concentration;
• diversity across relevant items or suppliers;
• downstream task success and return quality;
• latency, failure, and fallback rates;
• unequal quality or exposure across affected groups;
• support contacts and cases users cannot explain or correct.

Do not combine every concern into one score simply to make a launch decision easy. A guardrail exists to preserve a conflict the objective would otherwise erase.

Data-Informed Product Management provides a broader method for connecting decisions, evidence, metrics, and guardrails.

Plan for cold start and sparse evidence

New people, new items, and infrequent behaviour give the system little history. This is not an edge case; it is part of the product lifecycle.

Ask for a small amount of useful context when the benefit is clear. A chosen goal, role, or topic can provide a better start than silently inferring one from a few clicks.

Use item content, current context, editorial structure, or a segment baseline while personal evidence develops. Let people revise the initial choice.

Sparse evidence can also create a popularity loop. Items shown often collect more interactions, making them appear even more relevant and reducing the chance that alternatives gather evidence.

Measure exposure as well as response. Consider controlled exploration where it is appropriate, and preserve room for new or less-observed candidates.

Do not promise discovery while training the system to repeat the safest winner.

Give users control without making the product arbitrary

Control can improve the policy and the person’s mental model.

Useful controls include choosing a goal, following or muting a topic, hiding an item, resetting history, changing a profile assumption, or switching to a stable ordering.

Explain the local reason for a surprising recommendation when that reason helps the person judge it. “Because this project uses the same data source” is more useful than “Recommended by AI.”

Avoid pretending every visible control retrains a model immediately. State what changes now, what influences future results, and which effects apply across the account.

Consistency matters. If a personalised interface moves essential controls or makes support unable to reproduce the experience, the variation has created a service problem.

Keep critical navigation and account actions predictable. Personalise the choice where variation helps; do not personalise every part of the environment merely because the system can.

Evaluate offline, online, and over time

Offline evaluation can compare models on historical data. It is useful for rejecting weak approaches and checking technical behaviour.

Historical data was produced by the previous policy, so it cannot fully predict how a new policy changes exposure and behaviour.

Online experiments can estimate effects under real use. Define the eligible population, baseline, treatment, outcome window, and guardrails before interpreting a lift.

Check whether the effect comes from genuinely better choices or from novelty, a new layout, increased prominence, or more frequent exposure.

Evaluation continues after launch. Preferences change, catalogues shift, campaigns alter behaviour, input pipelines fail, and a once-useful label can become stale.

Compare new and established users, important contexts, and groups for whom an error carries different consequences. An acceptable average can hide a systematically poor experience.

Make the policy observable and supportable

A team should be able to reconstruct why an eligible person received a particular treatment without exposing sensitive model internals to every operator.

Record the policy and model version, candidate sources, applied filters, fallback, relevant input state, result, and later correction or outcome.

Monitor data coverage and freshness alongside response metrics. Silent loss of an input may leave the service running while the experience degrades.

Give support a safe way to identify the treatment a customer saw, explain available controls, and escalate a suspected failure.

Name owners for data inputs, policy rules, model behaviour, UX, and harmful incidents. “The algorithm did it” is not an operating model.

Decide whether to build, buy, or skip

Build when the policy is central to the product’s advantage, uses distinctive evidence, and requires control that a standard service cannot provide.

Buying can reduce infrastructure work for common recommendation, search, or messaging patterns. It does not outsource the product decision, data responsibility, evaluation, or recovery path.

Assess whether a vendor supports your eligibility rules, fallback, audit needs, data boundaries, experiments, and exit plan. A fast integration can become expensive when the product cannot explain or reproduce its behaviour.

Skip adaptive personalisation when the candidate set is small, the task is already clear, evidence is too sparse, the base experience is weak, or the expected value cannot justify the new system.

A well-designed contextual rule is a product solution, not a failed attempt at machine learning.

Treat consequential profiling as a higher-risk decision

Personalisation can range from ordering articles to changing access, price, employment, credit, insurance, or another consequential outcome.

The second category requires more than an improved recommendation UX.

The UK ICO’s draft guidance explains that, under UK GDPR, solely automated decisions with legal or similarly significant effects fall within specific ADM provisions.

Requirements depend on jurisdiction, purpose, data, process, and effect. Qualified privacy and legal specialists should assess the actual design.

Product teams should still ask early who is profiled, which inference is made, whether the person can understand and challenge it, and whether a less intrusive route can serve the outcome.

Do not turn a content-ranking playbook into permission to automate a consequential decision.

A hypothetical recommendation review

Consider a fictional analytics product that wants to recommend the next report an account administrator should configure. The scenario is invented to demonstrate the review.

The first proposal ranks reports by what similar accounts opened most. The objective is clicks.

The team examines the policy. New accounts have little history, account categories are broad, and opening a report does not mean the report was configured successfully.

Popular reports receive more exposure, while specialised reports rarely gather evidence. Some recommendations also require data sources the account has not connected.

The revised policy starts with eligibility. It removes reports that cannot work with the connected data and respects reports the administrator dismissed.

For cold start, it uses the account’s stated goal and current data context. The ranking objective becomes successful configuration, with clicks retained as a diagnostic signal.

Guardrails track repeated recommendations, fallback use, time to a trusted report, and configuration failure. Administrators can hide a suggestion and see the evidence behind an unfamiliar one.

The team compares the model with a contextual-rule baseline. It will adopt the model only if the improvement survives those product and operational costs.

The review has not asked whether personalisation is sophisticated. It has made the decision policy explicit enough to evaluate.

A practical personalisation review

Before building, ask:

• Which decision is personalised, for whom, and in what context?
• What user benefit should improve beyond a click?
• Would context, segments, or a rule solve the problem more clearly?
• Who is eligible, and which items or actions are never eligible?
• What does each input mean, and what happens when it is missing or stale?
• How are candidates generated, scored, filtered, and re-ranked?
• Which fallback preserves a useful product when the system is uncertain?
• What objective can the system learn, and which guardrails protect long-term value?
• How will new users and items receive a fair, useful start?
• Can people understand, correct, reset, or decline the treatment?
• How will offline evidence, online behaviour, and long-term drift be evaluated?
• Can support and product owners reconstruct a decision safely?
• Does building or buying outperform a simple baseline enough to justify the cost?
• Could profiling create a consequential legal, ethical, or access decision?

Personalisation is not the art of making every screen different. It is the discipline of choosing variation only where a defensible policy can improve somebody’s outcome.

Define the decision first. The right implementation may be a model, a rule, a segment, or the judgement to leave the experience alone.

Sources

• Recommendation systems overview — Google for Developers
• Rules of Machine Learning — Google for Developers
• Automated decision-making, including profiling — Information Commissioner’s Office

Read next

AI User Experience: Design for Judgment, Control, and Recovery examines how people understand, verify, and recover from intelligent product behaviour.