aifollow.news Search
Back Rohan Paul
Rohan Paul· @rohanpaul_ai · X· · Original publication time AI score54

Amazon research proposes checking what AI agents actually store in memoryMachine translation

Automatically verified and published · Generated and evidence-checked automatically; not reviewed by a human.

AI introduction

The study built FEB-30 from 30 scripted fact histories and 147 documents, with the correct memory state known after each document. In the memory system tested, stored-fact and question-answering scores differed by 16.8 points on average, and their pass/fail verdicts disagreed on 21% of fact histories.

Original post · Original

The article text is unavailable in this language; an existing version is shown.

@rohanpaul_ai

– https://t.co/cc0iI7sgEu

Title: "Beyond Downstream QA: Rethinking How We Evaluate Agent Memory"

ReplyRohan Paul@rohanpaul_ai
New Amazon paper shows that teams shipping agent memory should check what actually got stored, since QA scores alone did not reveal which pipeline stage was failing. Agent memory benchmarks grade the agent's answers, which mixes memory with retrieval and answer writing. A strong LLM can hide a broken memory store, and a retrieval miss can make a correct one look bad. They built FEB-30, 30 scripted fact histories turned into 147 documents, so the correct memory state is known after every document. On a memory system built like Mem0 and Hindsight, they compared the stored facts with that truth and with QA scores. The 2 scores differed by 16.8 points on average, and pass/fail verdicts disagreed on 21% of fact histories. The system picked the right memory action, like update or reinforce, only 20.1% of the time, which QA never showed. If you build agent memory, test the stored facts directly after each update, alongside end-to-end QA.
View this post on X ↗

来源:Rohan Paul · x.com

Research
Found an error? Send a correction