Incidents with AI agents: how to investigate, contain, and restore operations

An AI agent can consult data, select tools, update systems, and interact with other agents. When something goes wrong, the problem can quickly go beyond an incorrect response on the screen.

The agent can change a record, create a request, send a message, or repeat an operation. Depending on the permissions granted, a failure can produce effects in corporate systems and reach customers, teams, or critical processes.

At that moment, observing behavior is only the beginning.

The organization needs to limit impact, preserve evidence, understand what happened, correct the effects produced, and decide when operations can resume. That discipline is part of AgentOps, the set of practices used to operate AI agents in production with control, traceability, and defined responsibilities.

Responding to an incident involving agents requires addressing both the agent’s behavior and the actions it has already executed.

When a failure becomes an incident

Not every inadequate response needs to be classified as an incident.

An isolated error, identified before producing consequences, can be treated as a quality failure. The situation changes when there is real impact or relevant risk to people, data, processes, systems, or financial resources.

This can happen when the agent uses a tool outside the intended flow, accesses information incompatible with the task, repeats an operation, improperly changes a system, or forwards a request without preserving the necessary context.

There can also be an incident when the agent begins consuming resources abnormally, skips an approval step, or shows a persistent behavior change after an update.

Severity does not depend only on the number of errors. A single action can be critical when it involves sensitive data, financial movements, essential operations, or decisions with relevant consequences.

Therefore, classification must consider the scope of the action, the systems involved, the possibility of reversal, and the time available to limit impact.

Observability does not replace incident response

Observability makes it possible to monitor executions, model calls, tool use, errors, response time, and resource consumption. Evaluations make it possible to verify quality, adherence to criteria, and task completion.

These capabilities help detect the problem and provide evidence for investigation. However, they do not define who can suspend the agent, which permissions should be revoked, how to treat actions already executed, or who authorizes resumption.

Incident response begins when signals need to be converted into operational decisions.

That is what distinguishes this topic from a general discussion about monitoring. The goal is not only to see the agent’s behavior. It is to organize the company’s reaction when that behavior creates relevant impact or risk.

Containment can come before root cause

During an incident, there is a tendency to immediately seek the root cause. Investigation is important, but it must not allow the agent to keep producing potentially harmful actions.

Depending on the situation, the team can suspend new executions, block a tool, stop a queue, reduce permissions, or temporarily revoke the credentials used by the agent.

In environments with several agents and connected systems, dependencies must also be observed. Suspending one component can interrupt other flows. Therefore, the architecture must provide alternative paths, human review, or reduced operating modes.

AWS recommends combining behavioral signals with interruption mechanisms, credential revocation, and safe paths for dependent flows [1].

Containment must be fast, but not improvised. Possible actions should be foreseen in tested procedures and associated with people authorized to execute them.

How to organize the response

Although each incident has its own characteristics, a common sequence helps structure the response:

1. Declare and classify the incident. Confirm that there is relevant impact or risk, define severity, and identify who will coordinate the response.
2. Contain the behavior. Stop or limit actions that can amplify impact, preserving continuity for other processes as much as possible.
3. Preserve evidence. Record the execution, agent version, model, tools, permissions, calls made, and results produced.
4. Determine scope. Identify which tasks, users, data, and systems were affected and which actions were completed.
5. Correct the cause and address effects. Adjust the responsible component and execute the necessary corrections in external systems.
6. Validate recovery. Test the fix in controlled conditions, including scenarios similar to the one that produced the incident.
7. Resume gradually. Reactivate the agent with limited volume, permissions, or audience, and maintain reinforced observation during the initial period.

This sequence does not need to be fully linear. New evidence may require returning to investigation or expanding containment.

Preserving evidence without logging everything indiscriminately

Agents can use temporary context, memory, external tools, and different models during a single task. If the team simply stops the process, part of the information needed for investigation may disappear.

Preservation must allow reconstructing the sequence from the request to the final effect. This includes identifying which agent was running, its version, the configuration used, available tools, identity employed, and actions taken.

Calls, responses, human approvals, errors, repeated attempts, and timestamps also help explain what happened. That information needs to share a tracking identifier so it can be analyzed as a single execution.

OpenTelemetry’s generative AI conventions can represent model calls, agent invocations, and tool execution within distributed tracing [2].

However, messages, instructions, and tool arguments may contain sensitive information. The organization must define what will be collected, who may consult the logs, how they will be protected, and how long they will be retained.

Preserving evidence does not mean logging all content without criteria.

Investigation must follow the path of action

Investigating an incident with AI agents does not mean only asking why the model produced a given answer.

It is necessary to reconstruct the full path between the request received, the context used, the agent’s decision, the tool triggered, the action executed, and the result produced.

A failure seemingly attributed to the agent may have started elsewhere. The retrieved context may be outdated. A tool may have misinterpreted a parameter. A dependency may have responded incompletely. Excessive permission may have allowed an action that should have been blocked.

It is also possible that the agent behaved according to its configuration, but the configuration itself is wrong.

Investigation must consider the agent’s behavior, context quality, tools, identity used, permissions, orchestration, and external dependencies.

Recovery attempts must also be analyzed. A new attempt may resolve a transient failure, but it may duplicate an action when the previous operation completed and only the confirmation was lost.

AWS recommends classifying the failure before choosing recovery. Transient errors may allow a new attempt. Persistent failures may require an alternative. Unrecoverable situations must be forwarded to the responsible team [3].

An example in a business flow

Consider a fictitious agent responsible for preparing purchase requests for human review.

The agent consults order data, verifies registered suppliers, and creates a proposal in the corporate system. Final approval remains a person’s responsibility.

After a configuration change, some tool responses begin to take longer. The agent interprets the delay as failure and repeats the operation. Because the system lacks adequate duplicate protection, two proposals are created for the same request.

The first sign may be an increase in the number of calls. Then duplicate records and repeated notifications appear.

The team suspends new executions, temporarily stops the creation tool, and preserves traces. It then identifies every request processed in the period and separates attempts with no effect from actions that actually created records.

The fix involves adjusting handling of response times and implementing a key that prevents duplication of the same operation. Duplicate proposals need to be canceled or corrected before resumption.

This example shows why a technical change does not necessarily end the incident. In addition to correcting future behavior, the team must address effects that have already been produced.

Restoring a previous version does not undo external actions

The ability to return to a previous version is important. Prompts, tools, policies, models, and configurations must have identification and change history.

However, version rollback has a limit.

If the agent sent a message, updated a record, opened a ticket, or approved an operation, restoring its previous version does not automatically undo those actions.

The team may need to correct records, cancel requests, revoke access, reprocess tasks, or request new approvals. In some cases, the action cannot be fully undone, such as information already viewed or a message sent outside the organization.

Therefore, the architecture must consider compensating actions. Version rollback addresses the agent’s behavior. Compensation addresses effects left in other systems.

Resumption requires objective criteria

Pressure to restore operations can lead to premature reactivation of the agent.

Fixing a configuration and observing a successful test does not prove risk was controlled. Model-based systems can show behavior variations, and some problems depend on context or execution sequence.

Before resumption, the team must answer a few questions:

1. Was the cause corrected or is risk sufficiently limited?
2. Are permissions, tools, and configurations adequate?
3. Were prior effects that required correction addressed?
4. Did tests cover the scenario that produced the incident?
5. Are alerts, traces, and alternative paths working?
6. Did responsible areas approve resumption?

Microsoft recommends staged remediation, with immediate containment, expanded controls, cause correction, and observation periods after each stage [4].

Resumption can begin with reduced volume, limited users, temporarily restricted permissions, or additional human approval.

The goal is not only to put the agent back in operation. It is to restore operations to a known and controlled condition.

The procedure must exist before the incident

The team should not decide during a crisis who can suspend an agent, revoke a credential, or authorize resumption.

Those responsibilities must be defined in advance. The procedure should establish severity criteria, technical and business owners, communication channels, available containment actions, and conditions to resume operation.

It must also be exercised.

A controlled test can simulate tool unavailability, excessive permission, repeated action, or context loss between agents. The goal is to confirm whether technology and people respond as expected.

Preparation reduces time spent on basic decisions during a real incident.

AgentOps turns improvised reaction into operational capacity

As agents take on more relevant tasks, incidents cease to be only a development matter.

Operations must track versions, identities, permissions, tools, dependencies, evaluations, and traces. It must also relate these elements to actions executed in the company’s processes.

Observability helps detect and reconstruct behavior. Evaluations help measure quality. Incident response organizes the decisions needed when there is relevant impact or risk.

A prepared operation can identify which agent is involved, what changed, which actions were executed, who was affected, how to stop the behavior, and which conditions allow resumption.

Without these answers, the company may put agents into production but remain dependent on manual investigation and improvised decisions when something goes wrong.

Operating AI agents also means being prepared to interrupt, investigate, and resume their operation safely.

From incident response to operations with ARGUS

Responding to an incident involving AI agents requires reconstructing the relationship among context, decision, tool, action, and consequence. That investigation becomes harder when information is spread across agent platforms, models, APIs, corporate systems, and observability tools.

The ARGUS AgentOps proposal seeks to bring together visibility and governance mechanisms over agents’ actions. Traces, tool calls, errors, consumption, versions, and results help understand how an execution developed and where behavior deviated from expectations.

ARGUS also connects this view to operational context. Information about infrastructure, applications, services, events, and dependencies helps verify whether the incident began in the agent’s behavior or was influenced by a failure in the environment in which it was operating.

That correlation can support building a more complete timeline, identifying involved components, and providing evidence for investigation, supervision, and audit. It also helps monitor behavior after resumption, when operations must confirm the agent is functioning again within established conditions.

ARGUS does not replace process owners, containment procedures, or corrections needed in external systems. Its proposal is to offer the visibility and context needed for those decisions to be taken with better evidence.

Learn about ARGUS and see how it supports operations and governance for AI agents.

About Target Solutions

Target Solutions has worked for more than 15 years supporting organizations in evolving their infrastructure, monitoring, observability, automation, and operational intelligence operations.

That experience helps structure AgentOps practices aimed at visibility, governance, and operation of AI agents in corporate environments.

About the author

Paulo Florêncio Santos de Menezes is a partner and cofounder of Target Solutions, working in infrastructure, monitoring, observability, automation, IT operations, and AI agent governance.

He holds a degree in Data Processing from the Federal University of Amazonas, an MBA in Business Management from Fundação Dom Cabral, and a Professional Master’s in Economics from Fundação Getulio Vargas.

References

[1] AMAZON WEB SERVICES. Behavioral anomaly detection and agent containment. AWS Well-Architected Agentic AI Lens, 2026.

[2] OPENTELEMETRY. Semantic conventions for generative AI systems. OpenTelemetry Documentation, 2026.

[3] AMAZON WEB SERVICES. Agent monitoring, management and recovery. AWS Well-Architected Agentic AI Lens, 2026.

[4] MICROSOFT. Incident response for AI systems. Microsoft Learn, 2026.

[5] NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY. Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile. NIST AI 600-1, 2024.