An AI agent incident rarely looks like a traditional server outage. The model may still respond, the workflow may still complete, and every component may appear healthy while the agent sends the wrong message, changes too many records, discloses restricted data, or repeats an expensive action.
That makes a conventional uptime alert insufficient. Operations teams need a runbook that can answer five questions quickly: What happened? Is it still happening? How do we stop it? What did it affect? Under what conditions can it run again?
The runbook does not need to be long. It does need to exist before production access is granted, and it must cover the whole agent system—not just the model. The runtime, tools, identities, queues, approval service, data stores, and external destinations all belong inside the incident boundary.
This guide turns the governance controls described in our operating model for controlled AI agents into a minimum incident-response procedure.
Define an AI agent incident in operational terms
Do not wait for a perfect classification. Declare an incident when observed behavior exceeds the workflow's approved authority or risk tolerance. Useful triggers include:
- an attempted or completed action outside the allowlist;
- access to an unexpected data source, tenant, recipient, or destination;
- repeated tool calls, rapidly growing cost, or work continuing past its limit;
- a bypassed, mismatched, or missing human approval;
- exposure of credentials, personal data, confidential content, or system instructions;
- a material difference between the approved action and the executed action;
- evidence of prompt injection, compromised context, or a malicious tool response;
- unexplained changes in output quality, refusal behavior, or action patterns after a model, prompt, tool, or policy update.
NIST's AI Risk Management Framework calls for post-deployment monitoring that includes override, decommissioning, incident response, recovery, and change management. Translate that principle into measurable alerts. Monitor denied policy decisions, unusual tool-call volume, approval exceptions, new destinations, permission failures, token and monetary spend, destructive actions, and user-reported harm.
Each alert needs an owner, a severity, and a threshold. A denied attempt may be a low-severity signal. An unapproved external message, credential exposure, or destructive production change should trigger immediate containment.
Build a kill switch outside the agent
A prompt that says “stop” is not a kill switch. If the model is confused, manipulated, or operating on stale context, asking it to disable itself depends on the component you no longer trust.
The stop mechanism should live in deterministic infrastructure and be callable by the incident commander without model cooperation. At minimum, it should be able to:
- reject new runs at the workflow gateway;
- pause workers and cancel queued or scheduled tasks;
- revoke or disable the agent's service credentials and active sessions;
- block tool execution and outbound network destinations;
- prevent retries from recreating the incident; and
- retain logs and state for investigation.
Test these controls separately. A queue pause does not stop a worker already executing. Revoking an API key does not undo a message already sent. Disabling the model endpoint does not stop a deterministic retry processor. The runbook should name the control for every execution path and state who may activate it.
Use a graduated response where appropriate: pause one workflow, revoke one tool, switch the agent to read-only mode, or stop the entire agent platform. But provide a global stop for correlated failures. The architecture should already follow least-privilege, policy-enforced agent governance, because narrow credentials and allowlisted tools make containment faster and reduce the blast radius.
Contain damage before diagnosing the model
Once harmful behavior is credible, stop the action path first. Do not spend the first response window debating whether the cause was hallucination, prompt injection, bad retrieval, a tool defect, or operator error.
Assign one incident commander and record the declaration time, affected workflow, current severity, business owner, technical owner, and decisions. Then:
- freeze new agent runs and isolate affected workers;
- revoke workflow credentials or reduce them to read-only;
- suspend outbound messages, payments, deployments, and other consequential tools;
- quarantine untrusted prompts, retrieved documents, tool outputs, and memory entries;
- identify in-flight and recently completed actions;
- notify owners of affected systems so they can stop downstream automation;
- preserve volatile logs before retention windows or cleanup jobs remove them.
Containment should be idempotent: running the stop procedure twice must not restart work or destroy evidence. It should also fail closed. If the policy service, approval record, or destination cannot be verified, the action stays blocked.
CISA's incident-response playbook separates containment from eradication and recovery and recommends preserving evidence before moving forward. That distinction matters for agents. Deleting the suspicious conversation may remove the visible symptom while leaving poisoned memory, compromised credentials, queued actions, or a malicious document available to the next run.
Preserve a reconstruction, not just an error log
An application exception rarely explains an agent incident. Preserve enough information to replay the decision chain without executing it again:
- the user request, system instructions, retrieved context, memory, and relevant conversation state;
- model and prompt versions, sampling settings, safety and policy versions;
- every proposed and completed tool call, arguments, response, timestamp, and correlation ID;
- identity, permissions, credential version, and requesting user context;
- approval requests, the exact payload shown to the approver, decision, and approver identity;
- queue state, retries, timeouts, budgets, and external API responses;
- before-and-after values for affected records and a list of external recipients;
- deployment, configuration, model, tool, and knowledge-base changes preceding the event.
Protect this evidence from ordinary cleanup and restrict access because it may contain sensitive data. Hash or otherwise make critical exports tamper-evident, and keep an incident timeline. Follow the organization's legal, privacy, contractual, and notification requirements; the technical team should not improvise those obligations during the event.
Eradicate the cause across every layer
Only after containment and evidence capture should the team remove the cause. Work from the observed action backward through policy enforcement, tool implementation, orchestration, retrieved data, prompt, and model behavior.
The corrective action may be a narrower tool schema, a blocked destination, fixed tenant scoping, rotated credentials, removal of poisoned memory, a stronger approval binding, a retry limit, or a rollback of a recent configuration. A prompt edit alone is rarely sufficient when the incident demonstrated that infrastructure allowed an unsafe action.
Search for the same failure pattern in sibling agents and workflows. Shared tools, credentials, prompts, vector stores, policies, or model versions can turn a local incident into a platform-wide risk. Document the root cause and contributing controls that failed; avoid treating “the model hallucinated” as a complete explanation.
Recover with reduced authority
Recovery is a new production decision, not the inverse of the kill switch. Require approval from the business owner and the technical incident owner. Before restart, verify that:
- the unsafe path is blocked by a deterministic control;
- compromised credentials and untrusted context are removed or rotated;
- affected data and external actions are reconciled or corrected;
- the failure has a regression test using sanitized incident inputs;
- monitoring detects recurrence and routes to a named responder;
- rollback and stop controls have been exercised successfully.
Restore in stages: offline replay, sandbox, shadow mode without side effects, read-only production, a small allowlisted cohort, and finally the approved operating level. Start with lower rate, spend, and action limits. Increase autonomy only after an observation window produces clean evidence.
This mirrors a sound approach to AI agent orchestration: each component receives only the capabilities necessary for its step, and failure in one specialist does not silently grant broader authority elsewhere.
The minimum runbook to approve before launch
A production-ready agent should have a one-page operational record containing:
- incident triggers and severity thresholds;
- the incident commander and business, security, privacy, and system contacts;
- exact commands or console controls for pausing runs, queues, tools, credentials, and outbound traffic;
- the inventory of identities, tools, data sources, destinations, and downstream automation;
- evidence locations, retention periods, and access rules;
- containment and reconciliation checklists;
- recovery gates, approvers, and staged restart plan;
- communication and notification decision owners;
- the date and result of the last tabletop and kill-switch test.
Run a tabletop before launch and exercise the real stop path regularly. Test at least one in-flight action, one queued retry, one unavailable dependency, and one attempted restart with revoked credentials. Measure time to detect, time to stop new actions, time to identify affected records, and time to safe recovery.
The goal is not to predict every way an agent can fail. It is to make failure bounded, observable, stoppable, and recoverable under human control. If the team cannot demonstrate those properties, the agent is not ready for consequential production authority.
