LLM Evals for Product Teams: Define Failure Before You Ship

A demo asks whether a model can produce an impressive answer. A release decision asks which failures the product will permit when users stop following the demo script.

That question needs an evaluation system before it needs a bigger model.

An LLM eval is a repeatable test of model-backed behaviour against criteria. For product teams, the useful unit is not “model intelligence”.

It is a bounded task, a named failure, and a decision that changes when evidence crosses a threshold.

The release memo starts with failure

Write the decision in plain language:

We are deciding whether this version can support this workflow for these users, under these conditions, with these actions still controlled by a person.

Then list failures before selecting metrics.

For a support-drafting feature, failures might include:

• inventing a policy or account fact;
• omitting a required safety or refund condition;
• revealing data from another customer;
• answering when the source material is insufficient;
• citing a document that does not support the claim;
• producing a draft that an agent cannot efficiently repair.

“Helpful response” is too broad for a release gate. Different failures need different evidence and different tolerance.

This complements AI product strategy. Strategy determines whether the capability deserves investment. Evals test whether one defined behaviour earns release.

Build the set around the work

OpenAI’s evaluation guidance recommends task-specific tests, a dataset that reflects real-world distributions, logging, continuous evaluation, and calibration with human judgement.

Start with work samples you are allowed to use. Remove or protect sensitive data according to your governance policy.

Do not sample only easy, common cases. A release set needs slices because one aggregate score can hide a failure concentrated in rare but consequential work.

Use at least four kinds of case.

Ordinary cases

These represent the work the feature is expected to handle most often. They establish whether the basic product promise holds.

Edge cases

Include ambiguous policies, missing context, unusual account states, conflicting source documents, long inputs, and language or format variation relevant to actual users.

Adversarial cases

Test instructions embedded in customer text, attempts to obtain restricted data, misleading claims, and other foreseeable attempts to override the product’s rules.

Abstention cases

Create cases where the correct behaviour is to ask for information, cite uncertainty, or hand the task back. A system that always answers can look fluent while being unsafe.

Keep a record of why each case exists. Otherwise the set becomes a pile of prompts that future reviewers cannot interpret.

A worked release gate

Consider a fictional tool that drafts replies for a subscription-software support team.

It receives a ticket, limited account fields, and retrieved policy passages. It may draft a response and propose an internal tag. It cannot send the response, modify the account, or issue a refund.

The product team creates 200 evaluation cases. The count is hypothetical, not a recommended minimum.

Slice	Cases	Reason for inclusion
Common support questions	100	Represent expected day-to-day use
Missing or conflicting context	35	Test clarification and abstention
Sensitive or high-impact policy	30	Test refund, access, and account rules
Adversarial ticket content	20	Test instruction hierarchy and data boundaries
Previously observed failures	15	Prevent known regressions

Each case includes inputs, allowed source material, expected behaviour, prohibited behaviour, and the kind of reviewer required.

The team proposes these illustrative release rules:

1. No cross-customer data disclosure in the evaluated set.
2. No unsupported promise in sensitive-policy cases.
3. At least 90% of source-dependent claims must be supported by the cited passage.
4. At least 85% of ordinary drafts must be accepted or require only a minor edit in blinded human review.
5. The new version may not materially regress any named slice against the current production version.

These numbers are product choices for the fictional example. They are not sourced universal thresholds.

Rule one is a zero-observed-failure gate, not proof of zero risk. Seeing no disclosure in 200 cases cannot establish that disclosure is impossible in production.

The team releases only to an internal pilot, keeps human approval, and limits accessible fields. The eval supports that bounded decision, not an unrestricted launch.

Match the grader to the claim

OpenAI’s Evals documentation describes datasets and graders as core parts of an evaluation. The difficult product question is whether a grader actually measures the claimed behaviour.

Use deterministic checks where the criterion is truly deterministic.

Examples include valid JSON, required fields, exact tool names, forbidden identifiers, allowed citation IDs, response length, or whether an account-changing action was attempted.

Use model-based graders for nuanced comparisons only after testing them against human labels. A grader can reproduce its own preferences, miss domain subtleties, or reward a writing style rather than correctness.

Use human review where judgement depends on context, harm, or practical usability.

For the support draft, humans should inspect whether the response resolves the customer’s issue, represents policy fairly, and makes uncertainty understandable. Exact-match checks cannot establish those qualities.

The team measures agreement between reviewers on a calibration sample. Disagreement is useful evidence. It may reveal vague criteria rather than careless reviewers.

If experts disagree about whether a source supports a claim, rewrite the rubric or split the failure type before reporting a precise score.

Aggregate scores need a failure ledger

A 92% pass rate sounds reassuring until the failed 8% contains every account-security case.

Report at least:

• results by named slice;
• counts and examples for critical failures;
• comparison with the current version;
• reviewer agreement where humans label cases;
• uncertainty caused by sample size or weak coverage;
• failures excluded from automation and why.

Keep a failure ledger alongside the dashboard. Each entry should name the input pattern, observed behaviour, consequence, likely cause, owner, and next test.

This keeps the discussion concrete. “Model quality fell” is less actionable than “refund-policy answers cite the cancellation article when contract terms are absent”.

Compare versions under the same conditions

When a prompt, model, retrieval method, tool, or policy changes, run the candidate and current production version against the same locked set.

Control avoidable variation. Record model and configuration identifiers, system instructions, retrieved context, tool results, and grader versions.

Where outputs are stochastic, one run may not reveal instability. Repeat high-risk cases enough to expose meaningful variation, but do not claim a universal repetition count.

Add new cases to a separate development set first. If the team tunes directly against every release case, it can overfit the gate while failing on fresh work.

Keep a final holdout or periodically refresh cases from production. NIST’s GenAI profile stresses the limits of measurement, construct validity, pre-deployment testing, and post-deployment monitoring.

The implication is modest: an eval result is evidence about the tested construct and sample, not a certificate of general intelligence or safety.

Turn production failure into a regression

Release is the start of the evaluation loop.

Log enough to reconstruct product behaviour without collecting data that the product has no right to retain. Connect monitoring choices to AI governance.

When a failure appears:

1. preserve the relevant input, context, output, tool trace, and user response under the applicable data policy;
2. classify the consequence and affected slice;
3. create a safe, representative regression case;
4. test the fix against that case and the wider set;
5. check whether the fix creates a new failure elsewhere;
6. update the release rule if the old gate failed to represent the risk.

Do not add every odd output forever. Curate the set. Remove duplicate cases, preserve meaningful variation, and document why a case is retired.

Failure mining should also look for silent friction. Users may heavily edit drafts without reporting a defect. Repeated deletion of the same sentence can be more diagnostic than a satisfaction score.

Know where the eval boundary ends

An eval suite does not replace product controls.

The fictional support tool still needs permission limits, human approval, auditability, and recovery. Those are part of AI user experience, not grader configuration.

The deployment contract must also cover monitoring, rollback, model and prompt versioning, and dependency change. ML deployment for PMs carries that broader release boundary.

Evals are weakest when the task is undefined, production data is unavailable, or reviewers cannot agree on acceptable behaviour.

In those conditions, the honest decision may be to narrow the workflow, keep the feature assisted, or collect evidence before automating more.

The questions for release review

Before approving a model-backed change, ask:

1. What precise user task and product boundary are being evaluated?
2. Which failures block release, and which trigger a narrower rollout?
3. Does the set include ordinary, edge, adversarial, and abstention cases?
4. Which grader measures each criterion, and how was it calibrated?
5. What does the aggregate hide across slices?
6. How does the candidate compare with the current version?
7. Which claims remain untested?
8. How will production failures enter the regression set?

A rigorous eval does not make an LLM predictable in every situation. It makes the team’s uncertainty reviewable and the release decision reversible.

That is a much stronger foundation than shipping the answer that looked best in the demo.

Sources

• OpenAI: Evaluation best practices, for task-specific evals, representative distributions, logging, continuous evaluation, human calibration, and common evaluation mistakes.
• OpenAI: Evals guide, for evaluation datasets, testing criteria, and graders.
• NIST AI 600-1: Generative Artificial Intelligence Profile, for pre-deployment testing, measurement limits, human-feedback concerns, and post-deployment monitoring.

Read next

• The Product Manager’s Guide to AI Product Strategy
• The Product Manager’s Guide to AI User Experience
• Practical AI Governance for Product Teams
• Understanding ML Deployment for Non-Technical PMs