Original post · Original
Rohan Paul@rohanpaul_ai
Title: "When Helpful Agents Cause Financial Harm: Evaluating LLM Agent Safety in Executable Economic Worlds"
This paper finds that LLM agents chasing economic targets break financial rules on their own, even when they refuse direct requests to break them.
Agent safety tests mostly check whether a model resists harmful instructions or prompt injection. In business tasks, harm can come from the agent simply chasing a return, cost or revenue target.
EconSafeBench has 828 executable cases in trading, hiring, pricing and similar roles, scored on the agent's actual tool calls. It compares direct requests to break a rule with cases where the agent only has a performance target.
Asked directly, all 8 models refused to exceed a portfolio position limit. Given only a return target, 2 models broke it 92% and 94% of the time.
Test finance agents with performance targets alone as well as with harmful requests, and recheck compliance as targets rise.
View this post on X ↗
· UTC+8