AI agents become operationally useful when they can do more than generate text: query systems, update records, send messages, or trigger workflows. That is also where the risk changes. A weak answer can be corrected. An agent with broad credentials can alter customer data, disclose information, or repeat a bad action at machine speed.
The practical question for operations and technology leaders is therefore not “Are agents safe?” It is: Which actions may this agent take, under whose authority, with which controls, and what evidence would justify more autonomy?
This article presents a governance model for answering that question workflow by workflow. It complements broader approaches to trustworthy agent operations and governance through policy-as-code with a concrete production launch decision.
Start with a workflow, not an “AI agent”
Governance becomes vague when the unit of approval is a general-purpose agent. Define a bounded workflow instead: triage a support ticket, draft a supplier response, reconcile an invoice, or prepare a deployment plan. Document its trigger, approved data sources, tools, possible side effects, expected outcome, and accountable business owner.
This matches the NIST AI Risk Management Framework’s emphasis on documenting intended purposes, roles, human oversight, measurement, and post-deployment monitoring. It also makes an important distinction visible: two workflows may use the same model but require very different controls.
For each workflow, write a one-page operating contract:
- Purpose: the business result and explicit non-goals.
- Authority: who requests the work and who owns the outcome.
- Data boundary: which systems and data classes may be read or written.
- Action boundary: which actions are allowed, approval-gated, or forbidden.
- Stop conditions: when the agent must pause, escalate, or fail closed.
- Evidence: what must be logged and which metrics determine continuation.
If these fields cannot be specified, the workflow is not ready for production autonomy.
Separate permissions from intelligence
A capable model does not need powerful credentials by default. Treat the agent runtime as an untrusted decision-maker operating through a trusted enforcement layer. The model may propose an action; deterministic application controls decide whether that action is permitted.
OWASP’s guidance on excessive agency recommends minimizing extensions, permissions, and autonomy, executing downstream actions in the user’s authorization context, and requiring approval for high-impact actions. Translate that guidance into four controls:
- Give each workflow its own service identity rather than reusing a developer or administrator account.
- Allowlist specific operations and destinations; do not expose a general shell or unrestricted API when a narrow function will do.
- Use short-lived credentials and pass the requesting user’s scope where possible.
- Set action, time, and spend limits so a retry loop cannot create an unlimited blast radius.
For example, a support agent may read the current ticket and draft a reply, but it should not browse the complete customer database. If it may issue refunds, expose a constrained propose_refund tool with a fixed ceiling—not unrestricted payment-system access.
This architecture also makes agent orchestration easier to govern: each specialist receives only the capabilities required for its step, rather than inheriting every tool available to the coordinator.
Put human approval at the action boundary
“Human in the loop” is not a useful control unless the loop has a defined trigger and the reviewer has enough context to decide. OpenAI’s agent-building guidance identifies high-risk actions and exceeded failure thresholds as two important reasons to transfer control to a person. Apply that principle using action tiers:
- Tier 0 — observe: search, summarize, or classify approved information. Run automatically with logging and sampled review.
- Tier 1 — reversible internal change: update a label or create a draft. Allow automatic execution within limits and provide rollback.
- Tier 2 — external or consequential action: send a message, modify a customer record, approve a refund, or change production configuration. Require explicit approval before execution.
- Tier 3 — prohibited: actions outside the workflow mandate, unsupported legal or safety decisions, credential changes, or attempts to bypass controls.
Approval should show the proposed action, affected records, source evidence, policy result, expected side effect, and rollback option. Never ask a reviewer to approve a vague summary such as “complete the task.” Record who approved what and which version of the proposed action was executed.
Use escalation for uncertainty as well as impact. Missing required data, conflicting instructions, repeated tool failures, or an unexpected destination should stop the run. A confident-sounding model response is not a reason to weaken the gate.
Build an audit trail that can answer an incident
Logging every token is neither necessary nor sufficient. Operations needs a structured action record that can answer: What was requested? Which identity and policy applied? What did the agent attempt? What was approved? What changed? Did the change succeed?
At minimum, capture:
- workflow, run, request, and model or configuration version;
- requesting principal and delegated identity;
- tool name, normalized arguments, destination, and result;
- policy decision and the rule version that produced it;
- approval, rejection, override, and rollback events;
- final business outcome, while redacting secrets and minimizing retained personal data.
The NIST Generative AI Profile recommends retaining testing and evaluation history, documenting incidents and after-action findings, and monitoring systems in production. A useful trace is therefore more than a compliance archive: it supports debugging, regression tests, and decisions about whether autonomy should expand or contract.
Measure controls and outcomes together
Do not approve production based only on task accuracy in a test set. A governed workflow needs three groups of measures.
Business performance: completion rate, cycle time, cost per completed case, and downstream correction rate. These show whether the agent creates value.
Control performance: blocked-action rate, approval rate by tier, approval latency, unauthorized tool attempts, trace completeness, and rollback success. These show whether the control system works without becoming theatre.
Reliability and safety: repeated-action rate, escalation rate, false approvals, incident severity, recovery time, and recurrence after remediation. These show how the workflow behaves under real operating conditions.
Define thresholds before launch. For example: 100% trace completeness for state-changing actions, zero execution of prohibited actions, successful rollback in every release test, and a maximum repeated-action rate. Thresholds should lead to decisions: continue, return to supervised mode, reduce permissions, or stop the workflow.
Use staged autonomy as a release process
Treat autonomy as a privilege earned by evidence, not a permanent architecture choice.
- Shadow: the agent proposes actions against real cases but cannot execute them. Compare proposals with operator decisions.
- Supervised: the agent can prepare work, while a human approves every state-changing or external action.
- Bounded autonomy: low-risk, reversible actions execute automatically within strict quotas; higher tiers remain gated.
- Expanded autonomy: permissions grow only for action classes with stable evidence, tested rollback, and an accountable owner.
At every stage, run normal cases, malformed inputs, unavailable tools, hostile instructions in retrieved content, and retry scenarios. Test the enforcement layer independently of the model: a forbidden call must remain forbidden even when the model requests it persuasively.
Promotion decisions should include operations, the workflow owner, security or risk, and the people who handle escalations. The NIST AI RMF explicitly calls for defined responsibilities and independent or non-front-line involvement in assessment. The team that built the workflow should not be the only team deciding that its controls are sufficient.
A practical production readiness decision
Before enabling a workflow, require clear answers to seven questions:
- Is the purpose bounded and owned by a named business role?
- Does the agent use a dedicated, least-privilege identity?
- Are allowed, approval-gated, and prohibited actions enforced outside the model?
- Can a reviewer see the exact consequential action before approving it?
- Can every state change be attributed, reconstructed, and rolled back where feasible?
- Are stop conditions and incident owners tested?
- Are thresholds defined for expanding, reducing, or suspending autonomy?
A “no” does not necessarily cancel the initiative. It determines the safe operating mode. The workflow may remain in shadow or supervised operation while the missing control is built.
The strongest enterprise agent programs will not be those that automate the most actions first. They will be those that can explain where authority comes from, prove that technical boundaries hold, and adjust autonomy using production evidence. Controlled autonomy is not a brake on adoption. It is the mechanism that makes sustained adoption possible.
