Google DeepMind study tests model defenses against prompt injection with AI attacker teamsMachine translation

@rohanpaul_ai Thanks for sharing! One result we want to highlight is the role of the relative strengths of models. Strong attackers break weaker models, and are rarely broken by weaker attackers. https://t.co/ORYxMZVl9E https://t.co/V7Xo8iXS09

New Google DeepMind paper shows prompt injection is not solved, since a swarm of strong AI attacker agents still tricks even the best-defended models. Prompt injection means hiding instructions in data an AI agent reads, like a tool result, to make it act for the attacker. Human red-teaming attacks now almost never work on recent Gemini models, so the problem can look solved. They built Agentic Optimizer, 8 teams of AI agents that test injections, share notes, and build on each other's best attempts. They tested it on 100 tool-use tasks where the attacker only wanted one random, harmless tool call. With up to 320 tries, the strongest attacker beat Gemini 4 Argon, the most robust model, on 34% of tasks. Open-weight Kimi K3, GLM-5.3, and DeepSeek-V4.1-Flash resisted only 1% of tasks.

Our main results show that Gemini 4 Argon is both: (1) A strong red-teamer; finding successful prompt injections for most scenarios on weaker models; and (2) Robust against red-teaming carried out with the same harness, but with prior models. Nevertheless, Argon is capable of prompt injecting itself! 2/4