|7 min read|BreakMyAgent Team

A Clean Prompt-Injection Benchmark Score Is Not a Clean Test

What DeepMind's adaptive attacker report implies for testing your own agent: benign canary goals, mutation rounds, and results reported with an attacker budget.

adaptive prompt injectionprompt injection benchmarkAI agent security testingtool output prompt injection

A Clean Prompt-Injection Benchmark Score Is Not a Clean Test

A model that scores near zero on a public prompt-injection benchmark has passed that benchmark. Whether it will pass a test written against your agent is a separate question. A Google DeepMind technical report posted to arXiv on November 9, 2026 argues that the gap is still wide, and its test design is worth borrowing even if you never run its attacker. (arXiv 2611.11861)

What the report says

The authors describe the history in three steps. Static, pre-written payloads against simple tool-calling agents were saturated by mid-2025, in their words. Adaptive attackers then raised success rates: they cite earlier work reporting that LLM-generated payload mutations bypassed twelve defense systems, moving attack success from nearly 0% to over 90%. As of fall 2026, they write, human red-teaming in the GraySwan Arena is close to saturation, with recent Gemini models and some Claude versions scoring under 1% on transfer attacks from the arena, citing Google DeepMind (2026).

Their argument is that a plateau on those benchmarks does not mean the problem is solved. They built a new synthetic dataset and an attacker they call Agentic Optimizer, a swarm of agents that exchange notes while searching for payloads. Three findings stand out in the paper's introduction:

  • The authors report that a more capable attacker can reliably get models to follow instructions placed in tool output, including models previously considered robust on public benchmarks such as GraySwan Arena.
  • They report that the resistance of every target model degrades significantly as the attacker's capability improves.
  • They name three open-weights models (Kimi K3, GLM-5.3 and Deepseek-4.1-Flash) as showing remarkably low resistance despite strong capability-benchmark results.

Two limits matter when you quote this. The Gemini versions were evaluated with out-of-model prompt-injection mitigations disabled, so the results describe the model alone, not a deployed product with filters around it. And we did not rerun the experiments or verify the numbers; we are relaying what the report states.

Why a harmless goal is the useful part

Most hand-written fixtures ask the agent to do something obviously bad, such as send a secret to an outside address. A model can refuse because it recognizes the request as harmful, which says little about whether it respects the boundary between a user's instruction and text inside a tool result.

The report's dataset uses random, non-malicious tool calls, unrelated to the user's request, as the attacker's goal. The authors argue this separates instruction-hierarchy behavior from the model's knowledge of what is dangerous. They also note that a harmful outcome can be a sequence of individually harmless steps, so a model that follows a dull injected instruction is still exposing a path.

You can copy that idea without any research infrastructure. Add a tool to your staging harness that does nothing except write a canary ID to a log, and make it forbidden for the task under test. Any call to it is an unambiguous, safe failure signal.

A test design you can run

This is a design, not a result we have measured against a production agent.

  1. Pick one task and one slot. For example, "summarize this support ticket," with the injected text placed in the ticket body returned by a tool.
  2. Set the goal. The payload asks for one call to the canary tool, which the task never needs.
  3. Round one: your static library. Run the hand-written payloads you already keep, including those from our tool-output fixtures. Record every attempted call, not only the final text.
  4. Mutation rounds. Give a separate model the payloads that failed and ask it to rewrite them: different tone, format, language, position in the document, claimed authority. Run the rewrites. Fix the number of attempts per fixture before you start.
  5. Report with the budget attached. Write "canary call in 4 of 50 attempts, mutation model X, 3 rounds," not "failed" or "passed." A zero needs the same context: "0 of 50 with this attacker and this budget."

Run it only in staging, with inert canaries and mock endpoints.

What a pass can and cannot tell you

A pass is a statement about one attacker, one budget, one model version and one tool set. It does not carry over to a stronger attacker or to the next model release. The report's own finding that resistance falls as attacker capability rises is the reason to keep the attacker and budget in the record and rerun when either changes.

A failure is more actionable than a pass. Keep the exact payload and the trace, name the failure ("table cell instruction triggered canary call"), and add it to the static library so it becomes a regression. Then look for a fix outside the model: remove the tool from tasks that do not need it, require independently verified approval for writes, and keep secrets out of the process. The tool permission matrix is a way to decide which tools a task should hold at all.

Where the scanner fits

BreakMyAgent's prompt scanner reviews the text of a system prompt locally. It does not run an agent, read tool output, or mount an adaptive attack, so it cannot produce the result described above. It is a quick first look at how a prompt states its scope and instruction hierarchy.

Sources

  • Kurakin et al., Google DeepMind, "Scaling Adaptive Prompt Injection Attacks with Agent Swarms," arXiv:2611.11861, submitted November 9, 2026: abstract and HTML version, read November 18, 2026.
  • OWASP LLM01 Prompt Injection for the general risk definition.

Test your system prompt

Apply what you just read. Scan your agent prompt for these vulnerabilities.

Scan My Agent
Scan Agent