Carnegie Mellon and JetBrains study finds AI token spend is a poor measure of engineering outputMachine translation

New Carnegie Mellon and JetBrains paper finds that token spend shows how much engineers use AI agents, not how much they ship. Companies now track engineers' token spend to set budgets and judge productivity, and some even rank top users. No study had followed both spend and output for the same engineers over time. They linked 5 weeks of coding-agent token use to the commits of 404 JetBrains engineers. They also analyzed a company survey of 412 employees, where technical staff reported using agents beyond coding in nearly 9 of 10 evaluations. For the same engineer, doubling spend went with only about 10% more commits. Among weeks with similar commit counts, the 90th-percentile week used 39 to 240 times the tokens of the 10th-percentile week. Use token spend only for budgeting, and judge agent-assisted work by evidence about tasks and outcomes.