aifollow.news
Back Rohan Paul
Rohan Paul· @rohanpaul_ai · X· · Original publication time AI score54

Amazon Paper Studies How Agent Skills Go Stale and Proposes Selective MaintenanceMachine translation

AI introduction

An Amazon paper tested Claude Opus 4.7 and Haiku 4.5 before and after changes to their environments, finding that saved skill details can become a hindrance. Its method replays saved checks and repairs or rebuilds only conflicting content. On Terminal-Bench 2, a skill that had added 4.7 points for Opus left its pass rate 11.4 points below the no-skill result after the change.

Original post · Original

The article text is unavailable in this language; an existing version is shown.

@rohanpaul_ai

– https://t.co/XdlWSlNtqQ

Title: "When Agent Skills Go Stale: Runtime Behavior and Selective Maintenance in Evolving Environments"

ReplyRohan Paul@rohanpaul_ai
New Amazon paper finds that saved agent skills can hurt once the environment changes, and checking and repairing them selectively restores their value. Agent skills store past experience, like file paths, field names, or commands, so agents can skip work later. When an app or API changes, some of that stored detail goes stale without warning. They tested Claude Opus 4.7 and Haiku 4.5 on 4 settings, including Terminal-Bench 2 and IRS tax forms, before and after a change. Their fix replays saved checks witho1ut model calls and repairs or rebuilds only the conflicting skill content. On Terminal-Bench 2, a skill that had added 4.7 points for Opus cut its pass rate 11.4 points below no skill after the change. Maintenance lifted Haiku's tax-form field accuracy from 30.5% with the stale skill to 97.5%. Check saved agent skills against the current environment before reuse, instead of waiting for tasks to fail.
View this post on X ↗

Source: Rohan Paul · x.com

Research
Found an error? Send a correction