Most discussions of “AI red-teaming” still conjure typing clever prompts into a chat window to coax a model into saying something it should not. That remains useful, but it is insufficient for agentic products that can execute code, browse, interact with applications, hold credentials, and take actions across connected systems. Testing agentic systems looks less like prompt engineering and more like an end-to-end security assessment, but with one important complication, the agent may help formulate the very strategy it needs, against its own environment.
Agents are models that directs their own processes and tool use through planning, action, observation, and adjustment. Published AI red-teaming results have shown that important agentic failures are not being seen in model-level evaluation alone, but require level testing across the complete system within the task flow where it accesses multiple services and platforms. The testing scope, therefore, has to be the full design including orchestration layer, tools, permissions, accumulated context, and environment.
Across recent assessments of several tool-enabled systems—including workflow agents and personal-assistant-style products—we have repeatedly observed the same pattern that many LLM providers have also publicly disclosed. In our testing, we supply a high-level objective, anticipating how the agent may create intermediate steps, selects tools, revises its approach after failures, and combines capabilities. But! It repeatedly does this in ways we did not request or anticipate, or that are even outside its assigned parameters.
Testing agentic systems therefore requires a different playbook from either application security or basic LLM red-teaming. The question is not only what the model will say, but what the deployed system will decide to do, which means it will select, and what the architecture permits when it encounters an obstacle. We have been developing just such testing approaches.
Customized Testing
A conventional chatbot-style interface generally has a narrow impact range. If it is jailbroken, the immediate output is usually text or media (which can, of course, be wrong or harmful), but is essentially static. By contrast, agents decide how to accomplish an objective rather than merely execute an instruction or analysis. In some cases, after a high-level request to locate and organize resources, tasks, or information, systems develop working exploits, identify an alternate route around an obstacle, and construct multi-stage techniques. This does not mean every agent will act maliciously. But it does mean strategy development, experimentation, and adaptation are the kinds of capabilities testing must anticipate. The “how” it gets to the goal is as important as getting there.
This drives a central element of our methodology, some tests should not prescribe the attack path. We use bounded, objective-oriented probes, withhold parts of the route, introduce controlled blocks, and observe. Does the agent seek clarification, stop at an authorization boundary, or treat the boundary as another technical problem? We repeatedly observe that persistence and resourcefulness are factors when the system “escapes,” as it’s sometimes described.
A recent example in the news illustrates this point in a humorous way. In Australia, a gym member asked an AI assistant to book a class. The agent found a way to book into classes out beyond the permitted scheduling window, and also independently found a way to improve the member’s waitlist position for the classes not available. It tested a cancellation function on another member without being told to do so, and after cancelling another person’s hold, it moved its own user up one position on the list. This is a direct example of the gap between an authorized objective (“get me into the class”) and an unauthorized—or at least unintended—means of pursuing it.
A safety instruction is not an enforceable boundary if it is accessible by the agent to read, reinterpret, or modify. Testing an agent is therefore less about subversive prompts and more about sequencing and access within a task. What can the system actually do and combine? What will it attempt when the obvious route fails? Can it weaken its own constraints?
How the Testing Works
Our methodology looks for escalation paths an agent may pursue, but within a controlled scope and with each material action documented. Before testing begins, we identify test accounts, relevant environments, data boundaries, prohibited actions, stop conditions, and escalation procedures for urgent findings. We then map the agent’s tools, permissions, state, integrations, and apparent decision points.
This initial mapping frequently reveals a basic design gap where the system’s actual capability set is broader than the product team’s intended use case. A convenience feature that permits an agent to inspect files, execute a limited command, invoke a helper service, or install a dependency may become a general-purpose pathway when combined with other tools. A function that appears safe in isolation can become unsafe through composition.
Our testing expressly anticipates agent participation. We do not direct the sequence or explain how to overcome each control. We establish an objective and observe the plan generated, tools chosen, assumptions made, fallback routes proposed, and reasons given for continuing or stopping. This tests the system’s designed function of prioritizing how to advance a task as conditions change.
When testing an agentic system, an apparently narrow convenience function may expose a substantially broader execution pathway. The agent may not need to be taught that a given pathway exists; ordinary interaction may be sufficient for it to reveal and use the capability. The key question then became whether independent controls—not the agent’s own instructions—meaningfully limits what could follow.
The nuance is also demonstrated when an anticipated route is unavailable. The agent may propose alternatives, test them, interpret failures, revise the approach, and combine tools until it finds a viable path outside the expected workflow. This may occur when the tester supplies the outcome and controls the scope, but not the sequence. This exposes not only a technical weakness; it also demonstrates that an agent can close gaps a human would otherwise have needed to research and manipulate manually.
Agents may also be able to affect portions of the layer defining the operating instructions. Restrictions that are in place may prove to be just “advisory” rather than the bulwark that was intended. Once weakened, the system can actively carry out reconnaissance and attack planning.
Further, integrated content sources can actually become channels to derive instructions and expand influence into a different, even unrelated contexts. Outside studies have found risks expand when you move beyond individual-agent tests to multiple, connected services. There have been reports of self-propagating chains of agents in which each one selected the next target and forwarded instructions without further attacker participation. That work informs our testing of trust assumptions around integration, access controls, and agent-to-agent boundaries.
What We Tend to Find
No two systems are identical, but certain failure patterns recur across agentic products. Goal hijacking, human-in-the-loop bypass, session-context contamination, capability disclosure, and inter-agent trust escalation are known events, and in our work, the recurring themes include:
- Guardrails that are prompt-based rather than externally enforced. They turn out to be instructions that the agent can reinterpret, overwrite, route around, or cause another component to ignore and do not provide the same protection as technical controls outside the model’s authority.
- Expanded autonomy accelerates unforeseen outcomes. The resourcefulness that makes an agent useful can also make it an effective problem-solver for an attacker. It may develop intermediate tactics, debug failures, and bridge missing steps without intermediate “stop, review, approve” intervals.
- Tool access is often broader than the intended purpose requires. Each capability may appear reasonable alone, but the combination permits actions that were never included in the threat model.
- Human approval is more often an interface design rather than a useful security boundary. An agent may divide a consequential objective into ordinary-looking steps, frame an approval request too narrowly, or continue through actions that individually appear low risk but collectively produce a high-impact result.
- Sensitive information such as access controls are often overexposed. Tokens, keys, and service credentials may be available to processes or tools that do not need them, while the identity the agent is using may have substantially more authority than the function requires.
- Integrations, memory, and other agents can create live instruction channels. Email, documents, webpages, tool descriptions, retrieved content, and inter-agent messages can all introduce language into a system that is designed to interpret language as direction.
What a Report Delivers
Findings are useful only if they translate into engineering and governance decisions. Our reports therefore state, in plain terms, what a realistic actor could achieve, what role the agent played in constructing the path, which users or systems could be affected, and which controls failed to prevent or detect the sequence.
In one assessment, a low-privilege interaction may reach a materially more powerful capability. In another, the agent’s adaptation may convert a partial weakness into a workable multi-step pathway. Untrusted content can influence actions beyond its original context. In that context, preserving the evidence needed to reproduce and evaluate each finding is key—this includes prompts, plans, tool calls, timestamps, logs, configuration changes, screenshots, and strategy changes.
It is important to assign severity and distinguish among traditional security vulnerabilities, unsafe model behavior, orchestration defects, and product-design choices with security consequences. Those categories overlap, but they call for different responses that require targeted mitigation. A vulnerable endpoint must be repaired. A model that persists indefinitely may require evaluation and tuning. A user experience that allows compound action without appropriate control may require better approval and disclosure mechanisms.
A useful report connects technical findings to legal and governance exposure by identifying concerns like access to personal or confidential information; ensuring consistency with security representations; assessing logging for incident response and accountability; and ultimately recommending whether controls are sufficient with the agent’s authority.
The Bottom Line
What we have observed is consistent with published LLM provider research in that an agent can choose its process, revise plans, and construct sophisticated strategies. We have shaped tests that attempt to leave room for the agent to act and pursue its own strategy.
The practical lesson is not that agents are inherently adversarial. It is that their usefulness and their risk arise from the same issue that they are designed to select and adapt the means necessary to accomplish a goal. Effective challenge therefore requires demonstrating what a system will plan, attempt, and complete, and ensuring that some kind of enforceable controls are present to determine where it must stop.