Microsoft research finds coding agents struggle more with understanding code

Back Rohan Paul

Rohan Paul· @rohanpaul_ai · X· 2026-10-10 06:43 · Original publication time AI score54

Automatically verified and published · Generated and evidence-checked automatically; not reviewed by a human.

AI introduction

Microsoft researchers built CABRA to increase one kind of coding-task difficulty at a time, then tested 8 LLMs and 6 agents on 6,840 tasks. Agents stayed near-perfect as tasks grew by using tools such as grep. On SWE-bench Verified, reading and analysis call counts tracked failures more closely than lines edited.

Article · Original

The article text is unavailable in this language; an existing version is shown.

New Microsoft paper finds that coding agents trip up when they have to understand a lot of code, not when they have to edit a lot of it, so test them on reading and comparing code instead of diff size.

Microsoft researchers built CABRA, which generates synthetic coding tasks and raises 1 kind of difficulty at a time. They ran 8 LLMs and 6 agents on 6,840 tasks and labeled each tool call as reading, analyzing, searching, editing, or testing.

Plain LLMs got worse as tasks grew, but agents stayed near-perfect by using tools like grep. On SWE-bench Verified, the count of reading and analysis calls tracked agent failures better than lines edited, with correlations of -0.200 versus -0.159.

Found an error? Send a correction