When an AI Agent Makes a Mistake in Production, Which Layer Should Stop It?

The problem: a correct API call can still be a harmful action

Consider a support agent handling this request: a customer reports that a $12 add-on failed. The agent retrieves account data, decides compensation is justified, and calls a refund endpoint with the annual subscription line item instead. The refund API returns 200 OK because the subscription is refundable. The model followed its tool schema. Nothing technically crashed.

This is not primarily an output-quality problem. It is a control-boundary problem.

An LLM is useful for interpreting ambiguous language, extracting facts from a ticket, and proposing a next action. It is not a reliable enforcement point for financial limits, account ownership, product eligibility, segregation of duties, or irreversible side effects. A prompt saying “only refund the add-on” is a helpful instruction, but it is not a security boundary and it is not an accounting control.

The practical goal of an AI agent safety architecture is therefore not to make every model decision perfect. It is to ensure that an imperfect decision cannot become damage without passing the controls appropriate to its blast radius.

A useful design rule is:

The model may propose an action. Deterministic systems decide whether that action is authorized, valid, and safe to execute.

This tutorial builds that rule into a refund workflow. The same pattern applies to account deletion, access changes, order cancellation, database writes, production infrastructure changes, outbound email, and regulated decisions.

Assign each failure to the layer that can actually enforce it

Do not solve every failure with more prompting or a second model review. Start by asking what fact must be true before the action is safe, then place enforcement at the narrowest layer that has authoritative information and the ability to block execution.

Several layers can reject the same action. That is intentional. Defense in depth is valuable when the controls fail differently. For example, the policy validator can reject a $120 annual plan when the ticket refers to a $12 add-on, while the payment service can independently reject any request that lacks a capability scoped to the exact purchase line item.

Avoid a misleading version of defense in depth: calling the model twice and treating agreement as authorization. Two model calls can share the same missing context, misread the same ticket, or be manipulated by the same untrusted text. Independent enforcement requires independent data and deterministic rules.

Prerequisites

Before exposing any side-effecting tool to an agent, have these pieces in place:

1. An authoritative domain API for orders, subscriptions, payments, or the resource being changed. The agent should not infer identifiers from names when a backend can return exact IDs.
2. A stable action schema that represents a proposed action separately from execution. Use enums and IDs rather than free-text values for action types, currencies, and reasons.
3. A policy source of truth. Initially this can be version-controlled application configuration. Mature systems may use a policy service, but do not add one before you can state the rules clearly.
4. Reliable identity propagation. Record the end user, support operator or service principal, agent version, and workflow run ID.
5. A durable store for approval requests, action state, idempotency keys, and audit events.
6. An executor that supports idempotency. If the external payment provider supports idempotency keys, pass one through. If not, maintain a local deduplication record before invoking it.
7. A human escalation destination such as a case queue. Escalation must be a supported workflow result, not an exception path that silently drops work.

Do not start with autonomous refunds if you cannot reliably identify the purchase line item and customer. Start with draft-only recommendations or human-approved actions until the surrounding systems can supply those facts.

Step 1: Make the model produce a proposal, not a tool call

The model should return a narrow structured proposal. It must not receive a credential that can directly refund any subscription.

A proposal should contain the requested intent and the evidence the agent claims to have used. It should not contain an amount invented by the model. The amount, currency, product classification, ownership, and refundability should be resolved from domain records after the proposal is received.

type RefundProposal = {
  action: "REQUEST_REFUND";
  customerId: string;
  purchaseLineItemId: string;
  reason: "SERVICE_FAILURE" | "DUPLICATE_CHARGE" | "GOODWILL" | "OTHER";
  ticketId: string;
  evidence: string[];
  agentRunId: string;
};

The structured-output boundary is useful because it prevents malformed shapes from flowing downstream. It is not sufficient because a schema can validate a perfectly wrong ID. Treat schema validation as the first gate, not the safety system.

The orchestration service should validate that required fields exist, resolve the ticket and customer from trusted systems, and reject proposals referencing resources outside the ticket's account scope. If the request says “my add-on,” but the proposal selects a base subscription, proceed only if the policy layer can establish an allowed relationship.

Step 2: Rehydrate authoritative facts before policy evaluation

Never evaluate a financial action using values copied from the model response, retrieved text, or ticket body. Read authoritative records immediately before deciding.

For a refund, retrieve at least:

• the purchase line item and its immutable identifier;
• the owning customer account;
• product type and SKU;
• refundable balance after prior refunds;
• transaction status and any provider restrictions;
• currency and minor-unit amount;
• ticket identity and ticket-to-customer relationship;
• applicable policy version.

Use integer minor units for money. A $12.00 refund should be represented as 1200 cents, not a binary floating-point number such as 12.0.

type PurchaseLineItem = {
  id: string;
  customerId: string;
  productKind: "ADD_ON" | "ANNUAL_SUBSCRIPTION";
  refundableMinor: number;
  currency: "USD";
  status: "PAID" | "REFUNDED" | "VOID";
};

type RefundDecision =
  | { kind: "DENY"; reasons: string[] }
  | { kind: "REQUIRE_APPROVAL"; reasons: string[] }
  | { kind: "ALLOW"; refundMinor: number; policyVersion: string };

function evaluateRefund(
  proposal: RefundProposal,
  item: PurchaseLineItem,
  ticketCustomerId: string
): RefundDecision {
  const reasons: string[] = [];

  if (proposal.customerId !== ticketCustomerId) reasons.push("ticket customer mismatch");
  if (item.customerId !== ticketCustomerId) reasons.push("purchase is not owned by ticket customer");
  if (item.status !== "PAID") reasons.push("purchase is not in a refundable state");
  if (item.refundableMinor <= 0) reasons.push("no refundable balance remains");

  if (reasons.length > 0) return { kind: "DENY", reasons };

  // Example business policy: automatic goodwill is limited to add-ons at or below $12.
  if (proposal.reason === "GOODWILL" && item.productKind !== "ADD_ON") {
    return { kind: "REQUIRE_APPROVAL", reasons: ["goodwill refund is not an add-on"] };
  }
  if (proposal.reason === "GOODWILL" && item.refundableMinor > 1200) {
    return { kind: "REQUIRE_APPROVAL", reasons: ["amount exceeds automatic goodwill limit"] };
  }

  return { kind: "ALLOW", refundMinor: item.refundableMinor, policyVersion: "2026-01" };
}

The example deliberately uses a simple rule. Real policies usually include jurisdiction, payment method, cancellation date, fraud flags, tax handling, and contractual terms. Keep those rules in deterministic code or a reviewed policy system. Do not bury them inside prompts where they are difficult to test and audit.

Step 3: Convert approval into a state transition, not a chat message

When policy requires review, create a durable approval request containing a frozen action snapshot: purchase ID, authorized amount, reason, evidence references, policy result, and agent run ID.

The approver should see the authoritative facts, not merely the model's prose explanation. The approval action should approve or reject that exact snapshot. If the item, amount, customer, or policy version changes, invalidate the approval and create a new request.

A minimal state machine is:

PROPOSED -> VALIDATED -> APPROVAL_REQUIRED -> APPROVED -> EXECUTING -> EXECUTED
                         |                     |
                         -> REJECTED            -> FAILED
VALIDATED -> DENIED

Do not let a worker execute an action just because a ticket contains “approved” or because a reviewer left a comment. Approval must be a signed or authenticated state transition in your workflow store.

Use human approval for actions that are irreversible, high-value, legally sensitive, unusually frequent, or outside a policy's normal envelope. It is also appropriate for low-confidence evidence, conflicting records, and requests where a customer relationship exception requires judgment.

Step 4: Issue a short-lived, resource-scoped capability

The executor should not accept a general instruction such as refund(customerId, amount). That API gives the caller too much room to make a valid but harmful request.

Instead, after validation or approval, issue a server-side authorization artifact tied to one action. It should bind the action to the exact purchase line item, amount, currency, policy version, workflow ID, and expiry. The payment executor verifies it before doing anything.

Conceptually:

capability = authorize(
  action = refund,
  purchaseLineItemId = "line_123",
  amountMinor = 1200,
  currency = "USD",
  workflowId = "wf_456",
  expiresAt = short_lived_time
)

Whether this is a signed token or an opaque database record is an implementation choice. Opaque records are often simpler because revocation and state checks are straightforward. Signed tokens can reduce a lookup, but they introduce key management, claim validation, expiration handling, and revocation design.

The important property is scope: a capability authorizing a $12 refund for line_123 must be useless for a $120 annual-plan refund or for line_999.

Step 5: Make execution idempotent and append an audit record

Execution is where the external side effect occurs. It should receive only a validated command plus the scoped capability. Generate an idempotency key from a durable workflow action ID, not from model-generated text.

type AuthorizedRefundCommand = {
  workflowActionId: string;
  capabilityId: string;
  purchaseLineItemId: string;
  amountMinor: number;
  currency: "USD";
};

async function executeAuthorizedRefund(command: AuthorizedRefundCommand) {
  // 1. Atomically load and claim workflowActionId if it has not already executed.
  // 2. Verify the capability is active, unexpired, and matches every command field.
  // 3. Call the payment system using workflowActionId as its idempotency key.
  // 4. Persist the provider result and append an immutable audit event.
  // 5. Return the previously persisted result for a duplicate command.
}

The comments identify transactional requirements that must be implemented against your actual database and payment provider. Do not fake idempotency with an in-memory set: it fails during process restart, horizontal scaling, and message redelivery.

Your audit event should capture the proposal, authoritative IDs, policy outcome, approval identity when present, capability ID, executor request ID, provider response reference, timestamps, and software configuration identifiers. Avoid storing full ticket text or sensitive payment details unless retention policy explicitly permits it.

Step 6: Add observability and a circuit breaker for unsafe patterns

Operational telemetry should distinguish these outcomes:

• proposal schema rejection;
• ownership or entitlement denial;
• policy denial;
• approval requested, approved, rejected, or expired;
• capability validation failure;
• execution success, provider failure, and duplicate suppression;
• post-execution reversal or compensation.

A 200 OK from a tool is not an adequate success metric. Track whether policy blocks are rising for a specific agent version, prompt version, retrieval source, or tool schema. A sudden increase in invalid proposals can indicate bad context, changed product identifiers, a regression, or prompt injection attempts.

Open a circuit breaker on repeated semantic validation failures, not only HTTP failures. While open, disable autonomous execution for the affected workflow and route cases to approval or a static safe response. Retrying the same unsafe proposal with a different model does not fix a missing authorization boundary.

Common errors that leave a dangerous gap

Treating a system prompt as enforcement

Instructions such as “never refund more than $12” are not controls. The model can misunderstand, receive conflicting context, or encounter an adversarial ticket. Put the limit in policy code and at the executor boundary.

Giving the agent a broad production credential

A general payment API key means the model-adjacent process can perform any action permitted by that key. Use a brokered executor and issue narrowly scoped capabilities only after validation.

Validating the request but not the current state

A refund may be correct when proposed but invalid by the time it executes because another process already refunded it. Recheck mutable facts, especially refundable balance and action status, at execution time.

Asking an LLM to calculate money or select the final amount

Use the item record and policy engine to compute the amount. Models may identify the relevant item, but the final monetary command should derive from authoritative values.

Implementing approval as a notification

A Slack message, email reply, or ticket comment is not a safe approval mechanism. Persist an approval state tied to an immutable action snapshot and require authenticated approval.

Logging only prompts and model responses

Those records do not establish what actually changed. Log the complete control path: proposal, resolved resource, policy result, authorization, approval, idempotency key, and external transaction reference.

Rollout plan: reduce authority before increasing automation

Start in shadow mode. Let the agent create proposals while existing support processes continue to execute refunds. Compare its selected line item, reason, and policy result against real outcomes. This exposes ambiguous policies and data gaps without financial risk.

Next, allow the agent to create approval requests only. At this stage, reviewers can identify whether the model is consistently choosing the wrong resource or whether your domain data lacks a usable product mapping.

Enable automatic execution only for a small, explicitly bounded policy envelope, such as a refundable add-on below a defined limit with confirmed ownership and no fraud or dispute indicators. Everything outside that envelope escalates.

Finally, measure control effectiveness, not just automation rate. Useful measures include prevented invalid actions, approval overturn rate, duplicate suppression count, execution latency, post-execution correction rate, and the distribution of denial reasons. A high policy-denial rate may be a healthy safety control, a bad agent configuration, or a broken upstream data mapping; inspect it before trying to reduce it.

The safest production agent is not the one that appears most confident. It is the one whose authority is narrow, whose decisions are checked against authoritative facts, and whose side effects are impossible to perform outside a governed execution path.

FAQ

Should every AI agent action require human approval?

No. Require approval when the action exceeds a defined risk envelope. Low-risk and reversible actions can be automated if deterministic policy, scoped authorization, idempotency, and auditability are in place. High-value, irreversible, regulated, or ambiguous actions should escalate.

Can a second LLM act as the safety checker?

It can help identify ambiguity or classify evidence, but it should not be the only blocker for a consequential action. Use deterministic checks for ownership, product eligibility, amount limits, permissions, and current state. A second model is not an authorization system.

Where should business policy live?

Put enforceable rules in versioned deterministic code or a policy system that can be tested, reviewed, and audited. Keep the model informed about policy so it can make better proposals, but do not rely on the prompt as the authoritative implementation.

What is the minimum viable safe design for a refund agent?

At minimum: structured proposals, authoritative line-item lookup, ownership verification, deterministic amount and eligibility rules, a human approval gate for exceptions, idempotent execution, and an audit record. Do not grant the model direct access to a broad refund endpoint.

What if the payment provider cannot support idempotency keys?

Use a durable local action ledger with a unique workflow action ID and an atomic claim before calling the provider. Reconcile uncertain outcomes using provider transaction records before retrying. This is more operationally difficult than native provider idempotency, so treat retries conservatively.

Related Reading

• What the 100 Biggest GitHub Repositories Put in AGENTS.md Files
• Claude Code Review: Is Anthropic's Terminal AI Coding Agent Actually Worth It?
• Dify Ultimate Tutorial: Drag-and-Drop Enterprise Agent Workflows