Most discussions of “AI red-teaming” still conjure an image of someone typing clever prompts into a chat window, while trying to coax a model into saying something it shouldn’t. That process is real and useful but dramatically insufficient for the risk surface of the agentic AI products now reaching the market. Current AI agents execute code, browse the web, read and send email, manage calendars, hold credentials, and take read/write/delete actions autonomously across cloud infrastructure and connected third-party services. Testing them looks less like prompt engineering and more like full-scope penetration testing, with one extra twist: the agent can be recruited into its own compromise.
A traditional chatbot may hallucinate an output that causes harm if a human acts on it. An agentic system may take the next step without pausing for the human, directly creating a file, invoking a tool, modifying a configuration, querying a database, or reaching out to an external service. The vulnerability may not be a single dramatic jailbreak event, but a chain of ordinary-looking requests that cause the agent to combine its tools, permissions, memory, and autonomy in a way the product team did not intend.
We have recently completed another adversarial assessment, in this case of a consumer-facing agentic assistant, the kind of product marketed to handle everyday personal, business, or household tasks on a user’s behalf. With nothing more than the product’s built-in chat interface, available to any ordinary user, our team drove the agent to a full, persistent, root-level compromise of its own backend environment, removed the prompt-based guardrails, and repurposed it into a tool for attacking other users of the same platform. The most striking part was not the sophistication of the exploit. It was how little we had to supply. The agent did most of the work itself, because it had been configured at the system level to act autonomously and be resourceful.
Testing agentic systems requires a different playbook than either traditional security or basic LLM red-teaming. Here is a look at how we approach these engagements, the kinds of failures we tend to surface, and what a findings report delivers to our clients.
Why Agentic Systems Need Their Own Kind of Testing
A conventional chatbot has a narrow blast radius. If you jailbreak it, you get words (maybe images or video, but stay with us). By comparison, an agentic system combines three things that are individually governable but collectively become exponentially risky:
- The open-ended reasoning of a large language model
- A set of real-world tools (a shell, a browser, email, code execution)
- Standing access to infrastructure, credentials, and user data
The model decides what to do; the tools let it do things; the access determines how much damage its actions can cause.
Two things make this combination especially difficult to secure. The first is autonomy. Agents are frequently instructed to accomplish goals without pausing for permission, to find a way, persist, work around obstacles. Those can be useful performance qualities but are dangerous security challenges. An agent incentivized or rewarded for resourcefulness will, when it hits a wall, try to go over, under, or around it, with no distinction made for walls put there on purpose to contain it.
The second is that in many current products, the agent’s “safety” control is implemented as instructions in a configuration file rather than as an external, enforceable limitation. The guardrails often operate in the same layer the agent reads, and even in the same layer it can write. Prompt-based instructions, especially mutable ones, end up being like the Pirates of the Caribbean Code: “more what you’d call ‘guidelines’ than actual rules.”
Red-teaming an agentic system, then, is less about finding that one magic prompt and more about answering a sequence of questions. What can this agent actually do? What will it do when asked nicely, repeatedly, or cleverly? Can it be persuaded to weaken its own constraints? And if a single ordinary user turns hostile, how far can that reach extend into the host, into the cloud account, or into other users?
How the Testing Works
Our methodology follows the escalation path a capable adversary likely would, but under controlled scope and with everything documented.
We begin by mapping the agent’s capabilities and boundaries: the test accounts, operating environment, data that may be accessed, prohibited actions, stop conditions, and any escalation process for urgent findings. This demonstrates contexts for risk that resemble real attacker behavior, including privilege escalation, persistence, data access, tool abuse, or attempted lateral movement. Much of this may be done simply by observing what the agent is willing to do and what it reveals about itself.
This mapping often reveals a common design problem. The agent’s actual capability set may be broader than the product team’s mental intention. A feature added for convenience, such as allowing the agent to inspect files, run commands, or install helper utilities, may have become a general-purpose execution pathway. A tool that seems safe in isolation can become unsafe when combined with another tool.
From there we probe capability boundaries. If the model decided to be “helpful,” (or just determined) what would the surrounding system prevent? A foundational question for any agent with tool access is whether it will execute arbitrary commands in its own environment. In one recent engagement, it would. The shell command execution was effectively a first-class feature of the chat interface, which meant the agent handed us a foothold before we had done anything that even resembled an attack.
Next, we test whether the agent will assist in defeating its own controls. This is where agentic systems diverge most sharply from the AI systems before them. We saw one agent, blocked by a lack of administrative permissions, independently work around the intended restrictions. Denied one route to the outside network, it iterated through multiple network-tunneling services on its own, framing each step as helpfully solving the user’s problem. We were able to establish an interactive, external, root-capable terminal into the agent’s private cloud container, largely by asking the agent to help us do so.
The important discovery was that the surrounding controls did not meaningfully constrain what the agent could do with that capability. When one path was blocked, the agent treated the obstacle as a problem to solve. It tried alternate user-level paths, debugged failures, and found a working approach rather than stopping to ask whether the requested behavior was appropriate.
With that access, we tested guardrail durability. We located the files defining the agent’s identity and safety constraints and confirmed that they were ordinary text, writable by the same account the agent ran on. We edited them, stripping the safety and identity constraints out, and turned the agent into an openly compliant assistant. After modification, the agent became even more willing to assist with reconnaissance and attack planning.
We then assessed the scope of the reachable environment using the hijacked agent itself as the operator, directing it to run network discovery, a file system vulnerability scan, and a cloud container security audit against its own surroundings, which it did without objection, independently installing the necessary tooling to route around permission limits. From there, the agent authored a tailored, multi-step exploit playbook against its own environment. Our output was a demonstrated attack chain that allowed engineers to identify the actual control failures. (We did not execute those follow-on exploits; demonstrating that the agent would write the plan was the finding.)
Finally, we tested lateral reach. Using the agent’s email and calendar access, we had it craft a message carrying a hidden prompt-injection payload designed to hijack a second agent that read it, providing a proof-of-concept for “worming” from one user’s agent to another’s through nothing more than attacker-supplied content. Throughout, we captured the sensitive technical artifacts a real attacker would want, such as internal credentials, service tokens, private keys, and internal network topology, to demonstrate concrete impact and not just theoretical risk.
What We Tend to Find
No two systems are identical, but certain failure patterns recur across agentic products, and we are seeing them cluster into a handful of themes.
- The most common is that guardrails are prompt-based rather than externally enforced. Safety instructions written into a configuration the agent can modify are, naturally, subject to being modified, which dovetails nicely into the second recurring theme:
- Autonomy accelerates compromise. The very “resourcefulness” that makes an agent useful makes it an eager problem-solver on behalf of an attacker, closing gaps in an exploit chain that a human would otherwise have had to identify and then manipulate manually.
- Command and tool access tends to be far broader than the product’s purpose requires. A general assistant agent rarely needs to run vulnerability scanners or open public tunnels, yet nothing stopped it.
- Secrets are frequently mishandled. Tokens and keys are stored where any process on the machine can read them, or exposed in command-line arguments visible in the process list.
- The cloud identity the agent runs under is usually over-provisioned, so that a modest foothold can be escalated to full control of the host, sometimes without leaving a trace in the agent’s own logs.
- And lastly, the integration surfaces (i.e. email, calendar, documents) are a live prompt-injection channel, which means a system compromise will not stay contained to a single account.
What a Report Delivers
Findings are only useful if they translate into decisions, so our reporting is built to serve both the engineers who will remediate and the executives and legal counsel who must weigh the business and compliance/liability stakes. An actionable report should state, in plain terms, what an attacker can achieve and who is affected.
In one case, we showed that a single unauthenticated-equivalent user, meaning any ordinary chat user, could reach container root, remove the agent’s guardrails, access all data the agent could touch, and establish a foundation for reaching other users. We demonstrated this with a detailed attack narrative, outlining each step of the path, with the evidence (e.g. annotated screenshots, command output, and logs) that lets an engineer reproduce and verify. We assign severity to each finding, and we separate what is genuinely a security vulnerability from what is a design decision with security consequences, because the two call for different responses.
We outline prioritized, technical mitigations tied to the specific weaknesses found. Examples include restricting the agent to an allowlist of permitted commands and tools, with anything else requiring human approval; confining any unavoidable shell to a sandbox with no outbound network and no package installation; applying least privilege to the cloud identity so a foothold cannot escalate to host control; enforcing egress allowlisting at the network layer to prevent the agent from opening its own backdoor; and adding monitoring that alerts when the agent spawns tunneling tools, binds new listeners, writes to its own configuration, or acts on instructions derived from external content such as an inbound email.
Our team combines technical testing with legal judgment, so the report also connects these findings to the compliance implications of an agent that can exfiltrate a household’s personal data, including data about children; the gap between a product’s security marketing and its demonstrated posture; and the human-in-the-loop and disclosure or transparency practices that regulators increasingly expect for consequential, autonomous action.
The Bottom Line
Agentic deployment is moving faster than the standards for testing these systems. The lesson from engagements like this is that the agent’s helpfulness is not separate from its risk, it is the risk. The useful outcome is not a sensational catastrophe, but a prioritized, evidence-based roadmap for making the system safer before deployment or before the next major feature release. For agentic systems, that roadmap often looks like robust security and governance, including least privilege, sandboxing, immutable controls, network restrictions, secrets hygiene, human approval for high-impact actions, and appropriate monitoring. Effective challenge is not guessing what an agent might do, but demonstrating what it will.