Original post · AI translation
标题：“使用单一算法训练基础模型”
AI 训练的不同阶段可以合并为一种方法。
卡内基梅隆大学的一篇新论文表明，预训练、微调和强化学习可以共用同一套训练规则，而混合这些阶段的数据往往比将各阶段分开进行效果更好。
大语言模型通常分阶段训练：预训练、中期训练、微调，然后是强化学习，每个阶段都有自己的算法。阶段之间的突然切换也可能抹去先前习得的能力。
作者将一种名为 MaxRL 的强化学习方法重新表述为基于模型自身正确答案的似然训练。这样，同一个训练器就能随时间推移，以任意比例混合预训练文本、精选数据和强化学习样本。
在使用 124M 参数模型进行的推理谜题测试中，表现最好的混合训练运行结果超过了表现最好的分阶段训练运行结果，也更好地保留了通用能力。在随后进行编程强化学习后，一个 0.76B 参数模型在数学强化学习期间混入 5% 的中期训练数据，在 MBPP+ 上达到 32.69%，而分阶段训练的结果为 29.81%。
将训练流程视为数据混合比例随时间变化的安排，并尝试在强化学习中混入早期阶段的数据，而不是突然切换阶段。
Original
The separate stages of AI training can collapse into a single method. New Carnegie Mellon paper shows that pretraining, fine-tuning, and RL can share a single training rule, and blending their data often beats separate stages. LLMs are usually trained in stages: pretraining, midtraining, fine-tuning, then RL, each with its own algorithm. Hard switches between stages can also erase earlier skills. The authors recast a form of RL called MaxRL as likelihood training on the model's own correct answers. That lets one trainer mix pretraining text, curated data, and RL samples in any proportion over time. On reasoning puzzles with a 124M model, the best blended runs beat the best staged run and kept general skills better. After later coding RL, a 0.76B model that blended in 5% midtraining data during math RL reached 32.69% on MBPP+, versus 29.81% for staged training. Treat your training pipeline as a data-mixing schedule, and try blending earlier data into RL instead of switching stages abruptly.