AI User Experience: Design for Judgment, Control, and Recovery

An AI-generated summary can be fluent, useful, and wrong about the one detail that changes the decision.

The user then has two tasks. They must complete the original work and judge whether the system deserves reliance this time.

That second task is the centre of AI user experience.

The interface cannot make an uncertain system safe by looking calm or confident. It must help people form an accurate expectation, inspect what matters, correct the result, and recover when the system fails.

AI UX is ordinary UX with an unusual amount of uncertainty inside the interaction. The product still needs a clear job, feedback, control, error handling, and an outcome worth pursuing.

Start with the job and the possible harm

“Add an AI assistant” defines a technology and an interface shape. It says nothing about the work.

Begin with the moment in which a person is trying to make progress. What are they deciding, producing, finding, or changing? Which part is difficult, and why might an uncertain system help?

Then examine the cost of a plausible mistake.

A weak title suggestion may waste seconds. An omitted contract clause, incorrect medical interpretation, discriminatory ranking, or unauthorised account action can change somebody’s rights, money, safety, or access.

Severity is not the only factor. Consider who is exposed, whether the person can notice the error, how quickly it can be reversed, and whether another person bears the consequence.

The NIST AI Risk Management Framework treats risk as work across the AI lifecycle, not a test performed only before launch.

A UX review does not establish that a system is safe or compliant. It can reveal where the experience hides a risk, removes a safeguard, or asks a user to make a judgement without the information needed to do so.

For product-level governance and accountability, use Practical AI Governance for Product Teams.

Choose the system’s role explicitly

An AI feature can occupy very different roles in the same workflow:

• Draft: produce material for a person to review and complete.
• Suggest: offer one possible next action or value.
• Rank: order candidates while leaving the selection to a person.
• Decide: choose an outcome that the product applies.
• Act: execute one or more steps in the world or another system.

Risk and UX obligations change as the system moves down that list. A draft can still cause harm, but the user has a natural review point. An action may propagate before anyone sees its reasoning or result.

Do not describe an acting system as an assistant merely because the label feels friendly. Name what it can change, which boundaries it must respect, and where human authority remains.

The role should also be visible in the language of the interface. “Draft a reply” sets a different expectation from “Reply to the customer.” “Find likely duplicates” does not promise a complete or definitive list.

The AI-assisted workflow guide extends this decision across triggers, handoffs, approvals, and downstream actions.

Set expectations before the first result

People form a mental model from the entry point, label, empty state, example, and first output. A disclaimer below the result arrives too late to correct an inflated promise.

Explain the system in terms of the task:

• what it uses;
• what it produces or changes;
• which important limits apply;
• whether review is expected;
• what happens to the person’s data and feedback.

Avoid making the model category do the explanatory work. “Powered by advanced AI” tells the user less than “Creates a draft from the selected messages; check names, dates, and commitments before sending.”

Match the promise to observed performance in the relevant context. A feature that works well on short English support requests should not quietly imply the same reliability on every language, document, and account type.

Expectation-setting also includes availability. If the AI path depends on a service that may be slow, limited, or temporarily unavailable, preserve a usable non-AI route where the task requires one.

Show evidence and limits that support judgment

Explanation is useful when it helps someone decide what to do next. More detail is not automatically more transparent.

For a summary, the useful evidence may be links to the source passages. For a ranked alert, it may be the recent events that made the item relevant.

For a generated calculation, it may be the inputs and formula. For a suggested action, it may be the policy, constraint, and records the suggestion relies on.

Design the explanation around likely error detection:

• Which claim would be costly if wrong?
• What can the user verify quickly?
• Which source is missing, old, or ambiguous?
• Where has the system inferred rather than retrieved?
• What limitation changes how the result should be used?

A raw confidence score rarely answers those questions. Users may not know what the number measures, and a calibrated model probability is not the same as the chance that the complete answer is suitable for their task.

If confidence is shown, connect it to a decision and a reason. “Needs review: two records conflict on the renewal date” gives the user somewhere to look.

Do not use an explanation to make a weak result feel defensible. When the evidence is insufficient, the honest output may be no recommendation, a request for more information, or a manual path.

Support review, correction, undo, and escalation

“Human in the loop” is not a control if the person lacks time, context, authority, or a usable way to intervene.

Place review before the consequential action. Make changes visible. Preserve the original where comparison matters, and distinguish AI-produced content from source material.

Correction should fit the work:

• edit a draft in place;
• reject or replace a suggestion;
• change the source selection;
• restore the previous state;
• regenerate with a clarified constraint;
• send the case to a qualified person;
• turn the AI behaviour off where appropriate.

An undo control needs a meaningful window and recovery model. “Undo” after a message reached a customer or data reached another system may stop future processing without repairing what already happened.

Explain what feedback does. A thumbs-down may help a product team review quality, change a future preference, or become training data. Those are different promises.

Never imply that a correction has taught the model permanently unless the product can honour that behaviour and describe its scope.

Design more than one failure state

An AI result can fail without producing a technical error.

It can be wrong, incomplete, unsupported, stale, biased, inconsistent with policy, based on the wrong source, or correct but unsuitable for the current context.

The service can also be slow, unavailable, rate-limited, or unable to process the input. Each state changes the user’s next responsible action.

A useful failure design answers:

• What happened in language appropriate to the task?
• Did the system change anything before failing?
• Which work has been preserved?
• Can the person retry safely, edit the request, or use a manual path?
• Does a specialist need to review the case?
• How will repeated failure be detected by the product team?

Avoid turning unsupported output into a confident empty state. “No risks found” is a claim about the world. “The selected documents did not provide enough information to assess this risk” is a claim about the system’s evidence.

Aim for appropriate reliance, not maximum trust

Trust is often treated as a metric that should rise. That can reward an experience that makes an unreliable system look convincing.

The better objective is appropriate reliance: people use the system when it is useful and decline, verify, or escalate when the situation exceeds its capability.

Microsoft Research’s work on appropriate reliance describes the danger on both sides.

Over-reliance causes people to accept incorrect advice. Under-reliance causes them to reject useful support. Product design should help users discriminate rather than simply reassure them.

This changes what to test. Do users catch important mistakes? Do they reject correct output because the rationale is unclear? Can they tell when the task falls outside the supported scope?

Measure reliance against task quality, not agreement with the model. Disagreement may be exactly the behaviour a responsible product should enable.

Explain behaviour that changes over time

AI product behaviour can change because the model, prompt, retrieval source, policy, data, or user preference changed.

Silent changes damage the mental model people built through use. They can be especially serious when customers have reviewed, documented, or governed an earlier behaviour.

Tell users when a meaningful capability or boundary changes. Preserve version information where an output needs to be audited.

Reset an outdated preference rather than pretending it still has the same effect. Revisit examples and help content after model changes; stale documentation can be more misleading than no documentation.

Not every backend update needs a product announcement. Communicate changes that alter what users can reasonably expect, verify, control, or rely upon.

Turn feedback into a product and model loop

Feedback is useful only when somebody can interpret and act on it.

Instrument the workflow around the output: eligibility, request, source state, result, edit, rejection, action, undo, escalation, and downstream outcome.

Protect sensitive inputs and outputs. Collecting every prompt for possible future analysis can conflict with the purpose and expectations of the product.

Create an operational route for severe failures. A generic feedback queue is not enough for a harmful action, security issue, or systematic error affecting a group.

Combine explicit reports with behavioural and qualitative evidence. A person may silently rewrite a poor draft because reporting it costs more than correcting it.

Model-quality work and product-quality work need a shared record. The product team should know whether a failure comes from missing context, a poor interaction, a model limit, an unsafe policy, or the wrong use case.

Evaluate the whole human–AI task

An offline model score answers a narrower question than product success.

Evaluate the end-to-end task: time and effort, result quality, important errors detected, inappropriate acceptance, inappropriate rejection, recovery, accessibility, and the outcome after the person acts.

Segment the evidence by relevant task and population. An aggregate may hide failure on rare inputs, languages, accessibility needs, or high-consequence cases.

Microsoft’s human–AI interaction guidelines cover initial use, normal interaction, failure, and change over time.

They are a valuable review aid, not proof that a particular feature is trustworthy. Test the actual experience with people doing the actual work and with deliberately difficult cases.

Define launch guardrails and stop conditions. A faster average completion time should not excuse more consequential errors, inaccessible review, or actions people cannot reverse.

A hypothetical AI UX review

Consider a fictional support product that ranks incoming cases and drafts a reason for the priority. This scenario is invented to demonstrate the review.

The first proposal automatically moves predicted urgent cases to the top. Agents can see the priority but not the evidence, and the system learns from which cases they open first.

The team maps the failure. A missed urgent case delays help. A false urgent label displaces other work. Opening order is a weak label because agents already respond to the ranking.

The product role is changed from deciding the queue to suggesting a priority. It shows the recent account state and message phrases supporting the suggestion.

Agents can confirm, change, or escalate the priority. The original arrival order remains available, and the product records corrections separately from passive clicks.

The system abstains when the required account data is missing. A manual rule continues to surface cases with known safety terms, and a service outage restores the existing queue.

Evaluation includes time to appropriate response, urgent cases missed, false escalation, agent correction, and differences between customer groups.

The revised experience does not make the model correct. It makes its role, evidence, limits, and recovery compatible with the consequence of being wrong.

An AI UX launch review

Before release, ask:

• What job and decision does the feature support?
• Who experiences the benefit and who carries a possible error?
• Is the system drafting, suggesting, ranking, deciding, or acting?
• What expectation does the entry point set before the first result?
• Which evidence or limitation helps the user judge this output?
• Can the person review a meaningful change before it propagates?
• Can they correct, undo, opt out, or reach a qualified human path?
• What happens when output is wrong, incomplete, unsupported, or unavailable?
• Are we designing appropriate reliance or simply making the system look trustworthy?
• Which behaviour may change over time, and how will users learn about it?
• Does feedback have a named use, owner, and severe-incident route?
• Does evaluation cover the complete task and its unequal failure modes?

The strongest AI experience is not the one that makes the model disappear. It is the one that lets a person understand what role the system is playing and retain enough agency to use it well.

Design the judgment around the output with the same care as the output itself. That is where useful assistance becomes responsible product behaviour.

Sources

• Guidelines for Human-AI Interaction — Microsoft Research
• Fostering Appropriate Reliance on AI — Microsoft Research
• AI Risk Management Framework — NIST

Read next

Practical AI Governance for Product Teams connects product decisions about AI risk to ownership, evidence, escalation, and ongoing monitoring.

Responsible Product Management: Make Harm a Release Constraint extends that work beyond AI-specific controls into explicit harm ownership and release decisions.