原帖引用 · AI 翻译
标题：《解析强模型到弱模型的在策略蒸馏相对于 GRPO 的计算效率》
Salesforce 的一篇新论文发现，从大型外部教师模型进行蒸馏通常比普通强化学习耗费更多算力，而合并经过强化学习训练的领域专家模型可节省 42% 的算力。
在策略蒸馏让学生模型根据教师模型对其自身输出提供的逐 token 反馈进行学习，许多人预计这种密集信号会优于强化学习。但每个训练 token 还需要教师模型进行一次前向传播。
他们以每训练 FLOP 对应的留出集准确率为指标，比较了蒸馏与 GRPO，并改变了任务难度、模型规模、领域和训练设置。
使用经过后训练的外部教师模型时，GRPO 总体上表现更好，而蒸馏往往会使回答变长。将各领域的 GRPO 专家模型蒸馏到 1 个学生模型，在数学和代码任务上的表现与联合 GRPO 持平，而总 FLOP 减少了 42.0%。
为节省算力，先用强化学习训练各领域的专家模型，再将它们蒸馏到 1 个模型中，而不是从大型外部教师模型进行蒸馏。
原文
New Salesforce paper finds that distilling from a big external teacher usually costs more compute than plain RL, while merging RL-trained domain experts saves 42%. On-policy distillation lets a student learn from a teacher's per-token feedback on its own outputs, and many expect this dense signal to beat RL. But every training token also needs a teacher forward pass. They compared distillation and GRPO by held-out accuracy per training FLOP, varying task difficulty, model sizes, domains, and training settings. With an external post-trained teacher, GRPO generally won, and distillation often inflated response length. Distilling per-domain GRPO experts into 1 student matched joint GRPO on math plus code with 42.0% fewer total FLOPs. To save compute, train per-domain experts with RL and distill them into 1 model, rather than distilling from a big external teacher.