Original post · AI translation
标题：“被要求遗忘时，智能体能做到吗？大型语言模型智能体记忆遗忘基准”
清华大学的一篇新论文发现，当人们要求具有长期记忆的 AI 智能体忘记某项个人信息时，助手几乎从未能干净地将其删除：要么仍保留该信息，要么删除了过多内容。
具有长期记忆的智能体会存储个人信息，用户也越来越多地要求它们删除家庭住址之类的信息。现有的记忆基准测试只检验事实的存储和回忆。
他们构建了 BEMULA，包含 50 段较长的合成对话和 135 次遗忘请求，每段包含请求的对话都配有一个不含该请求的副本。探测测试检查相关事实是否已消失、其他记忆是否得以保留，以及攻击是否仍能提取该事实。
在 6 种记忆系统和 3 种基础模型的测试中，没有任何受测配置能端到端完成超过 1.5% 的遗忘请求。收到遗忘请求后，各系统最多只能拦截 35.7% 的简单信息提取提示，例如角色扮演或猜测答案是否为“是”的提问。
如果你的智能体会记住用户信息，就应建立并测试真正的删除流程，因为一条确认消息并不意味着数据已经消失。
Original
New Tsinghua paper finds that when people ask AI agents with long-term memory to forget a personal detail, the assistants almost never remove it cleanly, either keeping it or deleting too much. Agents with long-term memory store personal details, and users increasingly ask them to delete things like a home address. Existing memory benchmarks test only storing and recalling facts. They built BEMULA, 50 long synthetic chats with 135 forget requests, each paired with a copy that has no request. Probes check whether the fact is gone, other memories survive, and attacks can still extract it. No tested setup passed more than 1.5% of forget requests end to end, across 6 memory systems and 3 backbone models. After a forget request, systems blocked at most 35.7% of simple extraction prompts like role-play or yes-or-no guesses. If your agent remembers users, build and test a real delete path, since a confirmation message does not mean the data is gone.