aifollow.news
返回 量子位 AI 报道
量子位 AI 报道· · 原发布时间 AI 评分54

腾讯元宝与伦敦国王学院NLP团队推出群聊记忆评测基准

AI 辅助摘要

GroupAssistBench基于50个真实群聊的80,299条历史消息,构建2,395道题,分别考查AI助手能否回忆事实、在服务多人时用对不断变化的决定与偏好。14组配置的实验中,EverMemOS的记忆召回得分高于完整上下文,但开放式服务得分更低;论文、代码和数据集已公开。

正文 · AI 翻译

Submitted by Tencent Yuanbao × King's College London NLP team

QbitAI | WeChat public account QbitAI

A travel group of 9 initially agreed to leave on May 1, then changed the date to April 30. When someone asks an AI assistant to post a ticket-purchase reminder, it may still notify the group using the old date.

Although it has “remembered” the history, it has not actually served the group well.

Real group chats often involve subtler requirements: who wants a window seat, who has budget constraints, which arrangements are only proposals, and which have become shared decisions.

For an AI assistant serving a group, the key is not just retrieving memories but judging which person's memories should be used at this moment, and which ones.

To address this issue, Tencent Yuanbao and the King's College London NLP team jointly introduced GroupAssistBench. Based on 50 real group chats and 80,299 historical messages, the researchers created 2,395 evaluation questions to test whether agent memory can move from simply “remembering” to “using memories well.”

The research team compared 14 configurations, including full context, retrieval augmentation, and external memory. One representative contrast: EverMemOS scored 0.621 on the L1 memory recall evaluation, above the 0.570 scored by full context using the same task model; but on the L2 open-ended service evaluation, it scored 0.247, below full context's 0.431.

Being better at recall does not necessarily mean being better at providing service.

The paper is publicly available on arXiv, and the code and dataset have also been released.

△ Main experimental results for the 14 configurations: L1 memory recall is on the left, and L2 open-ended service is on the right. Comparisons should be made within each metric.

From a group chat to two tests: finding memories and using them correctly

Existing Agent Memory evaluations often test recall of past information through factual questions and answers, while recent work has also begun to examine downstream memory use. But real group chats pose more complex problems: different members may add to the same piece of information, plans change as discussions progress, and a single reply may be subject to constraints from several people at once.

GroupAssistBench builds its data from interactions with a group assistant deployed in real settings, preserving naturally occurring topic changes, changes in status, and interwoven information from multiple people. It divides the evaluation into two related task levels.

L1 and L2: from answering factual questions to providing a service

L1 memory recall has clear reference facts, such as “What departure date was originally agreed?” L2 memory use requires completing a service task, such as “Send everyone a travel and ticket-purchase reminder.” There is no fixed wording for the answer, but it must follow the current decisions and members' requirements from the chat history. The two categories contain 1,374 and 1,021 questions, respectively.

The study further divides questions by the memory structure they depend on: M1, a single memory; M2, integration of multiple memories; and M3, updates and conflict handling. M3 especially requires identifying when a new decision replaces an old one and which members' constraints still apply. These categories describe forms of information dependence, rather than a simple ranking by difficulty.

△ Overview of the evaluation framework: L1 factual recall and L2 open-ended service questions are built from the same group chat history and classified by whether they depend on a single memory, multiple memories, or updates and conflicts.

Every question must genuinely depend on the chat history

To prevent systems from guessing correctly using general knowledge alone, the team first compared answers with and without the chat history, selecting requests that genuinely required group chat memory. It then established Memory Anchors for the required facts, locating the source messages and evidence excerpts, and set a history cutoff for each question to prevent the model from seeing information posted after the question.

The data underwent anonymization and privacy review; the public version does not contain identifying information from the original group chats. The construction process combined programmatic rules, semantic review, and human verification. In a sample check, 95.8% of the questions were judged valid by majority vote.

△ Data construction process: select requests that genuinely depend on chat history, locate Memory Anchors, construct evaluation questions, and then check them through rules and human verification.

How do you score a service task with no standard answer?

When a group assistant helps plan an activity, reminds people to buy tickets, or coordinates disagreements, several responses may be appropriate. Simple string matching cannot determine whether the service actually satisfies the historical constraints. The team therefore created a customized rubric for each L2 question, tied to the corresponding historical evidence.

The rubric has three parts: Grounding checks whether the response explicitly reflects necessary historical facts; Service checks whether the task is completed; and Penalty penalizes errors such as violating current decisions, ignoring members' restrictions, or using outdated information. For example, a ticket-purchase reminder must use the new date while accounting for the relevant number of people, transport arrangements, and individual seat preferences. Treating a superseded date as the final plan should incur a penalty.

The design of Penalty also drew on real users' corrective feedback to the group assistant. The study used 1,152 criterion judgments across 60 L2 questions and 180 responses for human verification; automated judgments agreed with the human majority vote 89.3% of the time.

497 paired memories: why was a fact found but still not used?

To avoid mistaking differences between question types for differences in memory ability, the study tracked 497 paired memories: the same piece of history was directly queried in L1 and genuinely needed for an L2 service task. This made it possible to observe what happened during service to memories that could already be recalled.

Among the external memory systems analyzed, of the memories recalled in L1 but not judged to have been used effectively in L2, about 36% did not enter the L2 model input. The remaining roughly 64% entered the input but were not judged to have been applied effectively in the response. These percentages describe only what happened to this category of paired memories, not the failure rate across all L2 tasks.

The first case is related to requests lacking explicit retrieval cues: when a user says “Help everyone make arrangements,” they do not specify each historical preference that needs to be retrieved. The second case shows that giving the model evidence does not automatically mean it knows how that evidence should constrain its current reply.

△ What happened to the 497 paired memories: among memories successfully recalled in L1, compare whether they were applied in L2, entered the input but were not judged to have been applied, or did not enter the input.

Two interventions: connecting scattered evidence and turning historical requirements into constraints

Based on these findings, the team explored two lightweight interventions. RER (Relational Evidence Refinement) organizes the relationships among people, events, times, and decisions after retrieval, and adds relevant candidate evidence when needed. ACB (Assistance Constraint Binding) organizes obtained facts into response constraints with a subject, scope, and current validity status.

In the experiments, RER raised LightMem's average L2 score by 6.28 percentage points. ACB raised the scores of xMemory and EverMemOS by 3.94 and 3.16 percentage points, respectively. With both combined, the L2 scores of LightMem, xMemory, and EverMemOS rose by 7.03, 4.64, and 4.43 percentage points, respectively, relative to their own baselines.

These results suggest that organizing evidence and making constraints explicit may improve memory-driven service performance. The independent contribution of each mechanism still needs to be examined in more detail; the entire improvement cannot simply be attributed to “retrieving a few more memories.”

△ Effects of the RER and ACB interventions: comparing performance gains from organizing relational evidence, binding service constraints, and combining the two across different memory systems.

From remembering history to using it at the right time

Real group chats are a testing ground for personal agents and multi-user assistants. As agents take on more long-term service tasks, the value of a memory system depends not only on “how much it stores and how quickly it searches,” but also on whether it understands when to use information, whom it applies to, and which information is no longer valid. GroupAssistBench aims to make these differences observable, analyzable, and comparable.

Paper link: https:// arxiv.org/abs/2611.0486 7

Hugging Face paper: https:// huggingface.co/papers/2 611.04867

Code: https:// github.com/Tencent/Grou pAssistBench

Dataset: https:// huggingface.co/datasets /tencent/GroupAssistBench

Project homepage: https:// zhanghao-groupassisbench.github.io

发现内容有误?提交纠错

本文目录