aifollow.news
返回 Rohan Paul
Rohan Paul· @rohanpaul_ai · X· · 原发布时间 AI 评分54

Google 论文测试代理框架自我改进:分模块验证改动可控制令牌成本

AI 导读

研究让优化模型逐次修改一个 142 行编程代理框架的四个模块之一,并用验证集和令牌上限筛选改动。在 Gemini 3.8 Flash 的 40 道留出测试题上,这一方案解决率为 40.0%,每题使用 46.7k 令牌;不经检查的整体重写解决率为 42.5%,但每题使用 451.6k 令牌。

原帖引用 · 原文

该语言的正文暂不可用,当前显示已有版本。

@rohanpaul_ai

New Google paper finds that an unchecked full rewrite of an agent harness matched accuracy but cost nearly 10 times more tokens per task than gated modular edits.

that self-improving agent harnesses work best when they change 1 code module at a time and must pass validation.

The harness is the code around a model that manages context, runs tools, and checks work before stopping. A harness tuned for 1 model often stalls or wastes tokens on another.

They let an optimizer model improve a 142-line coding harness by editing 1 of its 4 parts at a time. Each edit had to pass a validation set and a token cap, or it was rejected.

On 40 held-out SWE-bench Lite tasks with Gemini 3.8 Flash, the gated harness solved 40.0% against 5.0% for the seed, using 46.7k tokens per task. A full unchecked rewrite solved 42.5% but used 451.6k tokens, and prompt rules dropped accuracy to 2.5%.

If you let agents tune their harness, gate each change on held-out tasks and token cost, and enforce rules in code, not prompts.

来源:Rohan Paul · x.com

论文
发现内容有误?提交纠错