Designing AI Agents: Set the Autonomy Boundary Before the Workflow

An agent is not a better form of every automation.

Sometimes the safest and most useful product is a rule. Sometimes a person should remain at the centre with strong assistance. Sometimes the system can choose and execute several steps within a bounded mandate.

The design decision is how much authority to delegate, under which conditions, and how a person recovers control when the system is wrong.

Choose that boundary before drawing the ideal workflow. Otherwise the happy path quietly becomes the permission model.

Three ways to handle the same work

Consider a fictional accounts-payable product handling supplier invoice exceptions.

An invoice may not match its purchase order. The product can retrieve records, identify likely causes, request missing approval, update an internal case, and eventually mark it ready for payment.

There are three materially different designs.

Design	What selects the next step	What the system may do
Deterministic automation	Explicit rules written in advance	Execute a fixed response when conditions match
Assisted interaction	Person, informed by model output	Suggest a diagnosis, message, or next action
Agent	Model-led orchestration within policy	Select tools and execute permitted steps towards a goal

The labels matter less than the authority. A chat interface that only drafts text is assistance. A background service that chooses tools and changes records has agentic behaviour even if nobody calls it an agent.

Start with the least autonomy that can solve the user problem. OpenAI’s practical guide recommends an incremental approach and distinguishes tools by factors such as read or write access, reversibility, permissions, and financial impact.

Option one: rules own the path

The invoice workflow can be deterministic when exceptions are well classified.

For example:

If the purchase-order currency differs from the invoice currency, place the case in manual review and notify the named owner using the approved template.

The rule is inspectable. Its inputs and output are defined. A product operator can test boundary cases without interpreting a generated plan.

Rules work well when the state space is narrow, actions are repeatable, and the cost of maintaining conditions is lower than the cost of model uncertainty.

They become brittle when exception language, document formats, and organisational routes vary faster than the rule set can represent.

Do not add a model merely because the rules are long. First ask whether the workflow itself can be simplified or the data contract repaired.

The existing guide to workflow automation helps design that deterministic path. The autonomy question starts when the system must interpret ambiguity or select among actions.

Option two: the person owns the decision

An assisted design retrieves the invoice, purchase order, policy, and approval history. It proposes a cause and drafts a request to the appropriate budget owner.

The operator reviews the evidence, edits the draft, and decides whether to send it. The product can make the work much faster without acquiring authority to act.

This is a strong default when:

• the interpretation is ambiguous;
• the action has financial or relationship consequences;
• important context lives outside the accessible systems;
• mistakes are easy for a person to spot before execution;
• learning about user judgement is still part of discovery.

Assistance is not automatically safe. Poorly presented evidence can cause automation bias, and repeated low-quality suggestions create review fatigue.

Design the review around the decision. Show sources, uncertainty, affected record, intended recipient, and the exact action.

AI-assisted workflow should make checking easier than doing the work again.

Option three: delegate a bounded goal

An agentic version receives a goal such as “resolve low-risk price mismatches under the approved tolerance policy”. It can inspect records, select a template, request approval, monitor for a response, and update the case.

This may help when paths vary but the action space can still be constrained. The system needs clear tools, state, exit conditions, and a way to recognise that its mandate has ended.

The word “bounded” does heavy work. The product team must specify what the agent can see, decide, change, communicate, repeat, and escalate.

Write the autonomy contract

For each action, record seven properties.

Property	Product question
Scope	Which goal and case types may the agent handle?
Permission	Which data and tools may it access?
Consequence	What can change for a user, supplier, or business?
Reversibility	Can the action be undone fully, partly, or not at all?
Confirmation	Who must approve, and what must they see?
Recovery	How is a mistake detected, stopped, and repaired?
Handoff	When and how does a person take over with context?

This contract should exist before implementation details obscure the decision.

For the invoice agent, the team proposes:

May read: invoice data, matching purchase order, approval chain, vendor master status, and the relevant policy version.

May write: internal case status, evidence notes, and a proposed discrepancy category.

May communicate: send an approved clarification template to an internal budget owner for low-risk mismatches.

Requires confirmation: any supplier-facing message, change to bank details, approval override, or payment-release action.

Must refuse: requests to bypass separation of duties, use unverified bank details, or reveal records outside the case.

Must hand off: conflicting source records, missing owner, repeated tool failure, policy ambiguity, or a value above the delegated threshold.

The threshold would be set from the organisation’s policy and risk evidence. This article supplies no universal financial limit.

Confirmation should match consequence

Do not put a confirmation box in front of every harmless action. People will learn to approve without reading.

Use consequence and reversibility to shape friction.

Action	Reversibility	Illustrative control
Retrieve records	No external change	Access control and audit log
Add an internal note	Easily corrected	Automatic with provenance
Email an internal owner	Partly reversible	Preview for novel content; template policy for routine cases
Contact a supplier	Social consequence	Human confirmation
Change bank details	Severe fraud risk	Agent prohibited; established verified process
Release payment	Financial consequence	Human approval and existing separation of duties

Confirmation is not a cure for broad permissions. An agent should not receive a powerful credential simply because a person clicks before the final call.

OWASP’s agentic guidance focuses attention on threats created by goals, memory, tools, identity, and multi-step action. It is a threat catalogue, not proof that one confirmation pattern makes an agent secure.

Bound retries and loops

An agent that fails to obtain approval should not keep messaging people until someone responds.

Set limits for tool calls, repeated messages, elapsed time, and spend. Define which failures may be retried and which require immediate handoff.

In the fictional workflow:

• one transient record-read failure may be retried;
• a second failure pauses the case;
• one reminder may be sent after the policy interval;
• no alternative approver is invented;
• the case expires into a human queue with the full trace.

Limits prevent a local error from becoming persistent nuisance or wider harm. They also make operating cost and latency visible.

The trace should show the goal, inputs, selected actions, tool results, policy version, confirmations, and final state. It should not store sensitive content indefinitely without a lawful need.

Connect retention, access, and review ownership to AI governance.

Design human takeover as a product state

“A human can intervene” is incomplete unless the handoff works under pressure.

The operator needs:

• a concise account of the agent’s goal and current state;
• source records and policy used;
• actions already taken and messages already sent;
• unresolved uncertainty;
• the next permitted choices;
• a control that stops further action.

Do not force the person to reconstruct the case from raw logs. Handoff should preserve enough reasoning evidence to act without pretending the model’s internal process is authoritative.

The user-facing experience also needs recovery. If the agent contacted a supplier incorrectly, the product must support correction and ownership, not hide the event behind an “AI generated” label.

That belongs in AI user experience: control, disclosure, correction, and recourse must fit the consequence.

A decision, not an autonomy ladder

No measured result is supplied for the fictional product. The following decision illustrates how the stated permissions and consequences could shape scope.

The team treats common mismatch categories as suitable for rules. Ambiguous exceptions need interpretation, but supplier communication and payment changes exceed what the stated controls can recover cleanly.

The chosen design is hybrid:

1. rules route known exceptions;
2. the model proposes a diagnosis and internal approval request;
3. the operator approves novel messages;
4. the system follows up once under policy;
5. payment release remains outside the agent’s tools.

The team will reconsider more autonomy after measuring correction rates, handoff causes, policy coverage, operator review effort, and incidents during a limited pilot.

This is not evidence that invoice agents are generally unsuitable. It is an honest decision under the stated permissions and recovery limits.

Review the boundary when the product changes

Autonomy is not a permanent property of a feature. New tools, data access, model behaviour, policies, user groups, or action values can change the risk.

Before expanding authority, ask:

1. Is the user problem unsolved by simpler rules or assistance?
2. Which new action is being delegated?
3. What permission makes it possible?
4. What is the worst credible consequence?
5. Can the action be reversed in practice, not only in theory?
6. Where must a person confirm or take over?
7. What trace supports investigation and redress?
8. What evidence would contract the boundary again?

An agent earns autonomy action by action. The product should never delegate more authority than it can observe, interrupt, and repair.

Sources

• OpenAI: A Practical Guide to Building Agents, for incremental orchestration, tool risk factors, layered safeguards, and human intervention for high-risk actions or repeated failures.
• OWASP: Agentic AI Threats and Mitigations, for threat modelling around agent goals, tools, identity, memory, and actions.
• OWASP Top 10 for Agentic Applications 2026, for a maintained catalogue of agentic application risks.
• NIST AI 600-1: Generative Artificial Intelligence Profile, for risk management across design, testing, deployment, monitoring, and incident feedback.

Read next

• The Product Manager’s Guide to AI-Assisted Workflow
• Practical AI Workflow Automation for Product Teams
• Practical AI Governance for Product Teams
• The Product Manager’s Guide to AI User Experience