正文 · AI 翻译
Long-video memory accuracy soars, with Q&A latency reduced by up to 97%
Yunzhong, reporting from Aofei Temple
QbitAI | WeChat public account QbitAI
Yunzhong, reporting from Aofei Temple
QbitAI | WeChat public account QbitAI
openJiuwen AgentMemory has officially launched its multimodal memory capability, OmniMem-Lens: blindly splitting videos by the second is the culprit behind fragmented memory; remembering by actual events is the right approach—recall accuracy improves by up to 45.7%, and online Q&A latency falls by up to 97%.
As AI agents and multimodal models (VLMs) evolve rapidly, we are no longer satisfied with having agents handle only text tasks; we want them to do more complex work.
For example, “Watch a 3-hour product launch and identify all the product highlights,” or “Analyze a full day of surveillance video and mark any unusual events.”
Yet when existing agents face very long videos, the sheer volume of information quickly exceeds their context limits, sharply reducing their ability to process the videos and their accuracy.
The prevailing industry practice is to crudely split long videos into short segments using fixed time windows, such as 1 minute, and have agents remember each segment separately.
This approach is like the blind men feeling an elephant, and it deals 2 fatal blows to an agent’s memory:
① Memory is chopped up—evidence becomes fragmented
An originally continuous explanation by a teacher is cut in the middle, and the agent processes and stores the first and second halves separately. When a user asks a question, the agent can retrieve only half the evidence, stripped of context, so it makes things up and frequently hallucinates.
② Memory is diluted—evidence gets buried
A crucial action lasting a few seconds is forcibly bundled with minutes of unrelated background footage before and after it. When the agent searches for details, the key information is buried in a mass of noise, and recall accuracy plummets.
The root of the problem is that agents remember video “by the second,” while people understand it “by the event.”
For example, when recalling a movie you watched yesterday, you remember plot points such as “the protagonists meet → misunderstand each other → reconcile,” rather than storing a segment every 60 seconds with a stopwatch. Yet today’s agents are effectively using a stopwatch to remember the movie.
For example, when recalling a movie you watched yesterday, you remember plot points such as “the protagonists meet → misunderstand each other → reconcile,” rather than storing a segment every 60 seconds with a stopwatch. Yet today’s agents are effectively using a stopwatch to remember the movie.
More troublesome still, most video event segmentation methods on the market rely on just 1 modality: they either look only at the images or listen only to speech. Both approaches can fail: when the image stays unchanged for a long time, a visual-only approach misses a change in topic; when there is no dialogue, an audio-only approach hears no clues.
The problem is not a lack of computing power; the memory loses its “focus” at the very point where it is segmented.
To enable agents to truly process long videos like the human brain, the team has officially launched OmniMem-Lens, a multimodal memory lens, in openJiuwen AgentMemory version 0.2.0.
As AgentMemory’s core technology for processing multimodal video memory, OmniMem-Lens takes an innovative approach: it understands events in videos through 2 tracks, visual and auditory, and organizes memory accordingly, allowing agents, for the first time, to truly store and retrieve video memories by event.
The team describes it as giving agents a “super hippocampus” that knows how to “focus automatically” and “organize in layers.”
It gets 3 things right: segmenting by actual events, storing memories in layers, and building memory offline for online retrieval.
Vision and hearing working together: remembering actual events like the human brain
OmniMem-Lens gives agents both “eyes” and “ears”: the eyes watch the images, detecting and marking changes when the scene shifts; the ears listen to speech, using ASR transcription to detect when a speaker changes topics even if the image stays the same.
By cross-checking these 2 forms of evidence, long videos are segmented at actual event boundaries, so the agent’s understanding of the video is no longer based on fragments taken out of context.
A 3-layer memory structure: storing different levels of detail separately for both detailed review and overall understanding
Each user question may require evidence at a different level of detail. OmniMem-Lens builds a 3-layer memory structure in the background:
It provides both outlines and details, retrieving whichever is needed without one interfering with the other.
It is like a book with a table of contents and an index: the table of contents (L1) takes you to the general chapter, the index (L2) pinpoints a specific sentence, and you can always return to the original text (L0) to check it.
It is like a book with a table of contents and an index: the table of contents (L1) takes you to the general chapter, the index (L2) pinpoints a specific sentence, and you can always return to the original text (L0) to check it.
At the same time, the L1 and L2 memory units are fully converted into text and vectors, allowing seamless integration with mainstream vector databases—developers with an existing RAG/Agent stack can use them with almost no modifications.
Build memory offline, retrieve answers online: do the time-consuming work in advance and give users fast responses
If every question asked from a different angle required the agent to watch several hours of video again, responses would be extremely slow and consume enormous computing resources.
OmniMem-Lens completes the heavy work of organizing memory offline in advance; when a question comes in, the agent looks up the memory directly, responding in seconds with an answer to the question asked.
The online retrieval process is straightforward: the user’s query is first semantically matched against L1/L2 memory units to quickly locate concise multimodal memories. Only when visual details clearly need verification does the system retrieve precise video frames from the L0 layer as needed. This architecture of a “lightweight text index + deferred retrieval of the original video” greatly improves the efficiency of video memory retrieval.
OmniMem-Lens is designed to pinpoint event boundaries with dual focus, like a professional camera, and arrange its memory structure in layers, like a prism refracting light, so that agents can move from blindly cutting up long videos to understanding them with precision.
Claims are one thing; the data tell the story. The team selected 3 mainstream long-video Q&A benchmarks and compared video-memory Q&A using OmniMem-Lens together with the Qwen3-VL-8B model against Q&A using the Qwen3-VL-8B VLM directly on video content.
Here is the table below in a nutshell:
Higher “recall accuracy” = more accurate evidence found;
Lower “online latency” = less time waiting for an answer.
Here is the table below in a nutshell:
Higher “recall accuracy” = more accurate evidence found;
Lower “online latency” = less time waiting for an answer.
① Memory recall accuracy: ahead on all 3 benchmarks
It improved by 8% on Video-MME-L, a general long-video benchmark, and by 6.4% on LVbench, a long-video understanding benchmark.
The standout result is +45.7% on EgoLife-QA. This benchmark records a full day of everyday life from a first-person perspective, with densely packed events and many small details—exactly the kind of setting where fixed segmentation is most likely to fail and where the value of “organizing memory by actual events” is clearest.
② Online Q&A latency: from “watch the video again” to “look up the memory directly”
The longer the video, the more frames a VLM answering directly has to work through, and the longer the delay. On LVbench, Qwen3-VL-8B takes 34.5 seconds, while OmniMem-Lens takes just 1.72 seconds—a full 20 times faster.
In everyday terms, a VLM answering directly is like rewatching a 3-hour movie from start to finish every time before answering you. OmniMem-Lens is like a film critic who took notes while watching: ask about a scene, and they check their notes and answer directly.
In everyday terms, a VLM answering directly is like rewatching a 3-hour movie from start to finish every time before answering you. OmniMem-Lens is like a film critic who took notes while watching: ask about a scene, and they check their notes and answer directly.
Example 1: The causal link is severed
Take a 2-hour animated film: at a station, a girl leaves and a boy watches her go. After she leaves, he imagines her turning into a butterfly and flying away.
Blindly cutting at fixed 1-minute intervals divides this 120-minute video into 120 clips. But the complete event—“the girl leaves → the boy watches her go → he imagines the butterfly”—runs from 14 minutes 20 seconds to 16 minutes 50 seconds, spanning the 14th, 15th, and 16th windows and being cut twice at the window boundaries.
The fragments then become 3 seemingly unrelated records: at minute 14, “the girl leaves”; at minute 15, “the boy watches her go”; and at minute 16, “the butterfly flies away.”
When a user asks, “What is the boy imagining after the girl leaves?”, the retrieved clips show only “the boy watches her go” and “the butterfly flies away,” with no clue that the butterfly is only imagined. The agent treats it as something that really happened and gives an answer that misses the point.
OmniMem-Lens instead stores the event as a single complete segment, from 14 minutes 20 seconds to 16 minutes 50 seconds: the ASR stream detects the topic shift, the visual stream captures the scene change, and the 2 streams jointly identify the actual boundaries. The causal link between “the girl leaves” and “he then imagines” is preserved in full, allowing the agent to distinguish fact from fantasy and answer accurately.
For the same video, the way memory is organized makes the difference between “unable to answer” and “answering accurately.”
Example 2: When the image is static, hearing is key
Visual clues alone can resolve the example above. In the following scenario, eyes alone are not enough.
In a 2-hour product review meeting, the screen shows the same architecture diagram throughout, with no change for 40 minutes. During those 40 minutes, however, the speaker covers 3 separate topics in turn: architecture design, performance stress testing, and next-quarter scheduling.
A visual-only approach, as well as blind segmentation into fixed windows, sees “1 static image lasting 40 minutes,” judges it to be a single stretch of content, and bundles all 3 topics into 1 memory entry. When a user asks, “What was the QPS in the performance stress-testing section?”, the key figure is buried in 40 minutes of noise and cannot be retrieved.
OmniMem-Lens’s ASR speech stream can detect that the speaker is changing topics even though the image has not changed. As soon as the speech shifts, the system makes a cut—each of the 3 topics becomes its own segment, making it easy to retrieve the right one when asked.
That is the value of the dual visual and auditory streams: when vision fails, hearing steps in; when hearing fails, such as during a scene change with no dialogue, vision steps in. Either stream alone can miss an event; cross-checking both makes the result reliable.
WorkSwarm will soon support integration with openJiuwen AgentMemory, helping you remember video content from everyday conversations and recall it quickly during discussions.
For agents to truly take on complex work for people, they need more than the ability to act; they also need a good memory.
By enabling agents to “remember by actual events” and “retrieve efficiently in layers,” OmniMem-Lens addresses fragmented memory, inaccurate retrieval, and slow responses in long-video scenarios. It provides a high-accuracy memory foundation with extremely fast responses for multimodal agents, enabling them to truly “remember everything they see and answer with ease” when handling long videos.
OmniMem-Lens has officially launched in openJiuwen AgentMemory version 0.2.0. You are welcome to try it and contribute.
If you found this article helpful, please like it, tap “Watching,” and share it. You are also welcome to discuss in the comments: how does your agent handle long-video memory?
Project address: https://gitcode.com/openJiuwen/agent-memory/
*This article is republished by QbitAI with authorization. The views are solely those of the original author.