aifollow.news
Back Rohan Paul
Rohan Paul· @rohanpaul_ai · X· · Original publication time AI score54

Google paper tests agent harness improvements through gated module editsMachine translation

AI introduction

An optimizer model edited one of four modules in a 142-line coding agent harness at a time, with each change screened by a validation set and token cap. On 40 held-out SWE-bench Lite tasks with Gemini 3.8 Flash, the gated harness solved 40.0% using 46.7k tokens per task; an unchecked full rewrite solved 42.5% using 451.6k tokens per task.

Original post · Original

The article text is unavailable in this language; an existing version is shown.

@rohanpaul_ai

New Google paper finds that an unchecked full rewrite of an agent harness matched accuracy but cost nearly 10 times more tokens per task than gated modular edits.

that self-improving agent harnesses work best when they change 1 code module at a time and must pass validation.

The harness is the code around a model that manages context, runs tools, and checks work before stopping. A harness tuned for 1 model often stalls or wastes tokens on another.

They let an optimizer model improve a 142-line coding harness by editing 1 of its 4 parts at a time. Each edit had to pass a validation set and a token cap, or it was rejected.

On 40 held-out SWE-bench Lite tasks with Gemini 3.8 Flash, the gated harness solved 40.0% against 5.0% for the seed, using 46.7k tokens per task. A full unchecked rewrite solved 42.5% but used 451.6k tokens, and prompt rules dropped accuracy to 2.5%.

If you let agents tune their harness, gate each change on held-out tasks and token cost, and enforce rules in code, not prompts.

Source: Rohan Paul · x.com

Research
Found an error? Send a correction