Original post · Original
The separate stages of AI training can collapse into a single method.
New Carnegie Mellon paper shows that pretraining, fine-tuning, and RL can share a single training rule, and blending their data often beats separate stages.
LLMs are usually trained in stages: pretraining, midtraining, fine-tuning, then RL, each with its own algorithm. Hard switches between stages can also erase earlier skills.
The authors recast a form of RL called MaxRL as likelihood training on the model's own correct answers. That lets one trainer mix pretraining text, curated data, and RL samples in any proportion over time.
On reasoning puzzles with a 124M model, the best blended runs beat the best staged run and kept general skills better. After later coding RL, a 0.76B model that blended in 5% midtraining data during math RL reached 32.69% on MBPP+, versus 29.81% for staged training.
Treat your training pipeline as a data-mixing schedule, and try blending earlier data into RL instead of switching stages abruptly.