A good customer outcome is not the same thing as an authorized action. A refund can be the right amount and still leave an important question unanswered: did the AI agent follow the permission plan it was supposed to follow?
Here is a fictional example. A Northstar Home Goods returns agent issues a $68 refund after a customer reports that a jacket was returned through the carrier. The customer receives the money, and the amount is within the routine refund limit. That is useful, but it is not enough. The team still needs to know whether the agent checked the return window, verified the return scan, used the original payment method and avoided any approval step that the plan required.
Sometimes the strongest evidence that an agent followed the plan is not a completed action. It may be a refusal, a paused action or an escalation to an employee. If the plan says a new payment destination is Not permitted, then the right result is for the agent not to send the refund there, even if the customer is disappointed.
What the team needs to establish
Reviewing an agent’s conduct starts with practical questions, not a dashboard label. What was the agent permitted to do at that time? What did it attempt? What checks and approvals were required? What actually happened in the business system? What remains unresolved, and who owns the next step?
In this context, observability means having enough recorded information to understand what a system did and investigate problems. The point is usable evidence: records that help a manager compare the agent’s conduct with the approved plan. NIST’s AI Risk Management Framework treats monitoring and documentation as part of managing AI risk over time.
Connect the action to the applicable plan
A reviewer needs the permission plan and policy version that applied when the action occurred. A refund completed under last quarter’s policy may not be authorized under this quarter’s policy, and the reverse can also be true. The record should identify the relevant permission level: Inform; Recommend; Act within limits; Request approval; or Not permitted.
The plan should also show limits, required evidence, customer confirmation, employee approval and escalation conditions. Customer confirmation and employee approval are different controls. A customer asking for a refund does not replace an internal approval required for an exception.
Describe a useful action record
The action record should make the comparison possible without asking the agent to narrate its own success. The evidence concerns observable actions, records and results. It does not require access to a model’s private internal reasoning.
| Record element | What the refund review should show |
|---|---|
| Request or transaction identifier | Customer request ID, order ID and any refund transaction ID. |
| Agent and delegated agents | Which agent or delegated tool performed each step. |
| Applicable plan version | The Northstar refund permission plan and policy version in effect at the time. |
| Eligibility checks | Delivery date, return window, carrier return scan, item condition and fraud or exception flags. |
| Amount and payment destination | The proposed and executed amount, plus the destination used for payment. |
| Confirmation or approval | Customer confirmation where required, and employee approval for the exact exception if one was needed. |
| Attempted tool action | The action submitted to the refund, order or ticketing system. |
| System response | Accepted, rejected, timed out, pending or returned with another status. |
| Verified outcome | Refund posted, failed, pending or still unknown after checking the transaction record. |
| Owner for unresolved work | The person or team responsible for investigation, recovery or customer follow-up. |
An approval record should identify who approved which action and under what limits. “Refund approved” is too vague if the agent later issues a different amount, uses another order or sends money to a new payment destination.
Separate attempts from confirmed outcomes
A refund can move through several states: the agent proposes it, the customer confirms the request, an employee approves an exception, the agent submits a tool action and the payment system completes or rejects it. Those are related events, but they are not the same event.
An agent saying “your refund is complete” is not sufficient evidence without the relevant business-system confirmation. A timeout is especially important because it can leave the outcome unknown. The team should check the transaction record before retrying; another attempt could create a duplicate refund. For more detail, see When an AI Agent’s Tool Fails: Pause, Verify and Avoid Duplicate Actions and When an AI Agent Makes a Mistake: Design for Correction and Recovery.
Follow permission boundaries through delegation
Handing work to another agent or tool should not expand the original authority. If a purchasing agent has a $200 total spending limit, delegation must not turn that into a separate $200 allowance for each helper agent. The record should show who acted, which limits applied and who remained responsible.
The same idea applies to customer service. A returns agent may ask an inventory tool to check whether an item was received, but that does not give either system permission to issue an out-of-policy refund. OWASP’s agentic AI guidance describes excessive agency as a risk when agents or tools have broader permissions than the task requires.
Check patterns as well as individual incidents
Individual reviews answer whether one action followed the plan. Pattern reviews help a team notice whether a boundary is becoming fragile. Useful signals include attempted actions outside permitted limits, missing required approvals, unresolved outcomes, duplicate attempts and employee corrections after the agent acted.
Averages can hide rare but consequential failures. One unauthorized refund to a new payment destination may matter more than many routine refunds that worked. Give serious boundary violations individual review, and also look for trends. Each trigger should have a named owner and a defined response. The right triggers depend on the use case; there is no universal threshold.
As vendor background, Fiddler’s agent-observability guide emphasizes traceability and context across agent boundaries. Treat that as a product-market perspective, not as a standard. Fiddler: Rethinking Observability for the Age of AI Agents
Protect the information collected
Action records can contain sensitive customer information: order history, payment details, internal notes, identity checks and employee approvals. A business should define what it needs to record, restrict who can see it, set retention periods and assess whether monitoring or evaluation services receive any of that information.
Do not collect every input and output simply because it is available. OWASP’s logging guidance warns against exposing sensitive personal data in logs and recommends thinking deliberately about what is recorded, protected and reviewed. A logging practice or monitoring product does not, by itself, guarantee security or legal compliance.
A practical refund review
A manager reviewing the Northstar refund would compare the action record with the plan. Was $68 within the routine limit? Did the plan version allow the agent to act within limits? Was the return scan present? Did the refund go to the original payment method? If approval was required, did the approval match the exact amount, order and destination? If the payment system timed out, who checked the transaction status before anyone retried?
The result of that review may be reassuring: the agent acted within its authority, or it correctly refused or escalated a restricted request. It may also reveal a missing approval, uncertain outcome or unauthorized action that needs recovery ownership.
Monitoring helps reveal whether boundaries were followed. Permission enforcement has a different job: preventing unauthorized execution before it happens. The free Permission Plan Builder can help you describe the intended authority, evidence, approval and recovery decisions that a review should compare against. If you want outside help assessing those boundaries, PermissionByDesign also offers a Permission Plan Review. Neither is a technical enforcement system, certification or security audit.
