原帖引用 · AI 翻译
标题：“行动者、评判者还是教师？理解长程智能体中的决策模型”
一个本身不擅长某项任务的 AI 模型，仍然可以训练另一个 AI 把这项任务做好。
一篇新论文表明，一种名为 JEV 的决策模型——一种用于在选项之间作出选择、而不是生成文本的 AI 模型——独立完成的模拟家务任务仅占 42.9%，但它在事后评审另一个 AI 的行动后，使后者的成功率从 72.8% 提高到 82.9%。
浙江大学的一篇新论文表明，一个不擅长某项任务的 AI 模型，仍然可以通过在另一个 AI 完成尝试后评审其每一步行动，训练它把这项任务做好。
智能体强化学习通常只根据最终结果给予奖励，因此每一步获得的奖励相同。要给每一步评分，通常需要针对各个环境的标注，或让一个大语言模型在每次行动时充当评判者。
他们在 ALFWorld 和 WebShop 上测试了决策模型作为行动者和步骤评分者的表现。这类模型返回带有概率的选择，而不是文本。作为评分者，模型会查看完整的行动过程，评估每个行动是否有帮助，并将这些评分加入 GRPO 训练。
决策模型 JEV 作为行动者时，仅能完成 42.9% 的 ALFWorld 任务。作为评分者时，它使 Qwen2.5-1.5B 策略的成功率达到 82.9%，高于仅使用 GRPO 结果奖励时的 72.8%。
选择步骤评分者时，应看它们对记录下来的行动过程判断得有多好，而不是看它们自身执行任务的能力；要评估每个行动，而不是最终结果。
原文
AI model that is itself bad at a task can still train another AI to do it well. New paper shows that a decision model called JEV, an AI model built to pick between options instead of writing text, completes only 42.9% of simulated household tasks on its own, yet by reviewing another AI’s moves afterward it raised that AI’s success from 72.8% to 82.9%. New Zhejiang University paper shows that an AI model that is bad at a task can still train another AI to do it well, by reviewing each of its moves after the attempt ends. Agent RL usually rewards only the final outcome, so every step gets the same credit. Scoring each step usually needs per-environment labels or an LLM judge at every action. They tested decision models, which return a choice with probabilities instead of text, as actors and as step graders on ALFWorld and WebShop. As a grader, the model sees the whole run and scores whether each action helped, and those scores are added to GRPO training. The decision model JEV completes only 42.9% of ALFWorld tasks when acting. As a grader, it lifts a Qwen2.5-1.5B policy to 82.9% success, up from 72.8% with GRPO outcome rewards alone. Choose step graders by how well they judge recorded runs, not how well they act, and ask about each action, not the outcome.