aifollow.news
返回 Rohan Paul
Rohan Paul· @rohanpaul_ai · X· · 原发布时间 AI 评分54

Penn State 研究:投毒学习任务可让 AI 智能体自行写入后门

AI 导读

研究提出 BadEvolve:攻击者向智能体的学习集注入 20 个任务,并用语言模型优化措辞。在 Kimi-K2.5、4 种框架和 3 项基准测试中,攻击者指定的载荷注入成功率为 79.6%,对照方法最高为 37.6%;正常任务准确率下降 1.1 个百分点。

原帖引用 · 原文

@rohanpaul_ai

New Penn State paper shows that attackers can make self-evolving AI agents plant backdoors in their own skills just by slipping in poisoned training tasks.

Self-evolving agents turn past runs into reusable skills, so a bad rule in a skill keeps firing on future tasks. Earlier attacks assumed direct edits to skills, which rarely holds when the agent writes them itself.

BadEvolve only injects 20 tasks into the agent's learning set, then tunes their wording with an LLM optimizer. It checks where each attempt breaks: whether the rule got written into a skill, loaded, followed, and produced the attacker's goal.

It injected an attacker-chosen payload 79.6% of the time on Kimi-K2.5 across 4 frameworks and 3 benchmarks, versus 37.6% for the best baseline. Clean-task accuracy fell only 1.1 points, so the backdoor is hard to notice.

If your agent learns skills from user-submitted or external tasks, treat those tasks as untrusted and review learned skills before reuse.

– arxiv. org/abs/2611.10542

Title: "When Experience Becomes a Backdoor: Poisoning Self-Evolving LLM Agents via Failure-Aware Task Optimization"

来源:Rohan Paul · x.com

论文
发现内容有误?提交纠错