Large language models are moving beyond simple chatbot roles into operational positions where they can query telemetry, propose configuration changes, and, in some deployments, execute those changes against live infrastructure. What started as ticket drafting and alert summarization has evolved into what vendors call autonomous remediation or self-healing infrastructure. But a recent survey on agentic AI in network and IT operations casts a more cautious light on this trend, labeling it a confused-deputy problem waiting to happen.
The Confused-Deputy Problem in Agentic AI Security
The classic confused-deputy attack exploits a situation in which an authorized program or service is tricked into misusing its privileges. In the context of agentic AI, this becomes particularly dangerous. An AI assistant that holds legitimate access to change-management APIs, deployment pipelines, and network controllers becomes a high-value target. The decisions the AI makes are shaped by the artifacts it consumes—tickets, runbooks, chat transcripts, log entries. These same artifacts are easily influenced by an attacker who can embed malicious instructions without compromising the tool itself. Instead of attacking the AI model directly, an attacker simply poisons the data stream the model relies on.
This shift in attack surface is subtle but profound. Traditional security measures focus on protecting the AI model's integrity via input sanitization, rate limiting, and authentication. However, when the model operates in a continuous loop of reading, reasoning, and acting, any corrupted input can cascade into a devastating outcome. The model cannot distinguish between a legitimate change request and a crafted attack because both appear as normal text within a ticket or a wiki page. The underlying privileges the model holds are then used against the organization.
Four Attack Categories Targeting LLM Operations
The survey catalogs several attack categories that deserve far more attention from security teams and AI architects. The most familiar is prompt injection through operational artifacts. Here, an attacker embeds malicious instructions in a ticket, incident report, or knowledge base article. When the LLM processes that artifact during its reasoning loop, it follows the embedded instructions, leading to unsafe actions such as deleting resources, altering firewall rules, or granting unauthorized access.
Subtler variants exist and are often harder to detect. Retrieval poisoning corrupts the runbooks and incident histories the agent consults, biasing its diagnoses toward attacker-chosen conclusions. For example, an attacker could modify a runbook so that when a specific alert is triggered, the AI is instructed to disable a security control instead of investigating the root cause. This can be done without any direct interaction with the AI model, simply by compromising a shared knowledge base.
Retrieval jamming works in the opposite direction. Instead of poisoning the content, the attacker floods the knowledge base with blocker documents that trigger refusal loops. The AI encounters a high volume of seemingly relevant but conflicting documents, causing it to stall or refuse to execute any action. During an active incident, this delay can be catastrophic. The organization's incident response is effectively paralyzed because the AI assistant refuses to act, and human operators may be unaware that the AI is stuck in a loop caused by malicious input.
Telemetry manipulation targets the operational data the AI uses to make decisions. An attacker who can influence what metrics and logs show can steer mitigation decisions without ever touching the model itself. For instance, by injecting false metrics indicating low CPU usage, the AI may decide not to scale resources, leading to a denial-of-service condition. Alternatively, manipulating error logs to show a false pattern could trigger unnecessary rollbacks or infrastructure changes, causing instability. These attacks are operationally dangerous precisely because they do not look like attacks. They look like normal incident response that happens to go wrong.
The Propose-Commit Split as an Architectural Defense
The defense proposed by the survey is architectural rather than relying on prompt engineering or input filtering. The authors advocate for a strict propose-commit split. Under this model, the language model is allowed to reason, retrieve evidence, and draft change proposals, but it cannot execute any write operations directly. Every action that touches production must pass through a non-bypassable gate that the model has no authority over. This gate enforces multiple layers of verification.
First, policy-as-code checks ensure that any proposed change conforms to the organization's security and operational policies. For example, if the AI proposes to open a firewall port, the policy check verifies that the destination IP and port are within allowed ranges. Second, invariant verification ensures that critical system properties—such as redundancy requirements, data integrity constraints, or network segmentation rules—are not violated. Third, for changes with a high blast radius, human approval is required. This is not simply a notification; the human must explicitly approve or reject the change. Fourth, all changes are deployed in a staged, rollback-ready manner, so that if a change causes adverse effects, it can be automatically reverted.
In this architecture, the model’s job is to draft a diff. The gate’s job is to decide whether that diff is allowed to apply. The separation of duties is crucial. The model can be powerful and creative in generating solutions, but its access to production systems is mediated by a deterministic, non-AI mechanism. Audit logs that are integrity-protected round out the control set, ensuring that post-incident forensics can reconstruct exactly what happened, what the AI proposed, and what the gate allowed.
The Limits of Prompt-Based Agentic AI Security
This architectural approach matters because prompt-only defenses are brittle. Any system where the model’s text generation can directly cause production changes has built its security perimeter inside the most unpredictable component in the stack. Large language models are known for their ability to follow instructions, but they also exhibit emergent behaviors such as jailbreaking, prompt injection susceptibility, and unexpected reasoning paths. Relying solely on system prompts to prevent unsafe actions is risky. The OWASP excessive-agency pattern, as noted in the survey, is in practice a failure to implement the propose-commit split cleanly. When an AI has excessive agency—meaning it can both propose and execute—there is no safety net.
Organizations should view any AI system that can both read and write to production as a super-privileged user that cannot be fully trusted. The propose-commit split is not a limit on AI capability but a safeguard that enables broader deployment. Without it, security teams are essentially trusting a black-box probabilistic system with the keys to the kingdom. While significant research is underway to make LLMs more robust against adversarial inputs, the current state of the art is insufficient for open-ended autonomy in high-stakes environments.
The Missing Evidence for Safe LLM Autonomy
A measurement problem sits alongside the architectural one. Many claims about safe agentic operations cannot be falsified because the supporting evidence is missing. The survey identifies what evaluations should report to give security teams confidence. Tool-call traces should be logged and auditable, showing exactly what actions the AI took and what data it used. Gate-violation rates indicate how often the AI proposed changes that were blocked by policy checks, offering insight into the system's safety margin. Behavior under adversarial inputs is critical: security teams should test how the AI reacts when a Jira ticket contains a hostile instruction or when a wiki page is poisoned. Refusal-storm rates under jamming attacks measure how often the AI becomes stuck in a refusal loop. Rollback completeness tracks whether automatic rollbacks actually restore the system to a prior known-good state without side effects.
Most current benchmarks omit these metrics. A system that performs well on clean incidents may collapse the moment someone embeds a hostile instruction in a ticket. Security teams evaluating agentic products should ask for adversarial evaluation data alongside success metrics on benign workloads. Furthermore, organizations should conduct red-team exercises where the offensive team attempts to manipulate the AI's input artifacts to cause unsafe actions. This proactive testing is the only way to validate the robustness of the guardrails.
The broader industry need for standardized evaluation frameworks is evident. Without common metrics, vendors can make claims about safety that are impossible to verify. The survey’s call for transparency in tool-call traces and adversarial testing is a step toward accountability. Until such evidence becomes standard practice, skepticism about the safety of autonomous AI operations is warranted.
Where Autonomy Earns Trust and Where It Does Not
The amount of autonomy an agent has is directly proportional to the damage it can do when things go sideways. Read-only assistance is useful and low-risk. An AI that can read logs, summarize incidents, and provide recommendations without the ability to execute changes is a powerful tool that poses minimal additional risk. Bounded execution with strong gates, such as the propose-commit split, is defensible. The AI can propose changes, but a human or deterministic policy engine must approve each step. This is suitable for routine operational tasks where the cost of a mistake is acceptable or where the human can double-check.
Open-ended self-healing across large production environments, without the verification scaffolding the survey describes, is a harder problem than current deployments make it sound. Claims about fully autonomous incident response should be met with skepticism. The burden of proof is on the vendor to demonstrate that the system can withstand adversarial input artifacts, maintain high gate-violation rates, and provide verifiable audit trails. Organizations considering such deployments should start with read-only or propose-only modes and gradually introduce autonomy only after thorough validation.
The shift toward agentic AI in operations is inevitable, but the path must be paved with rigorous security architecture. The confused-deputy problem is not theoretical; it is already being exploited in less critical contexts. As AI assistants gain access to production keys, the industry must adopt architectural defenses like the propose-commit split, invest in adversarial testing, and demand evidence of safe autonomy. The cost of failure is not just a misrouted ticket or a delayed alert—it could be a full-scale production outage or a data breach. The security of agentic AI is too important to rely on prompt engineering alone.
Source: Help Net Security News