Deep dive A · post-training · source snapshots 21 and 22 September 2026

How production recipes sequence SFT, preference tuning, RL and on-policy distillation

Post-training is no longer one stage: documented pipelines chain SFT, preference optimization, verifiable-reward RL and, increasingly, on-policy distillation, in orders that differ by lab. This page records each recipe in the source's own words, the frameworks per sub-stage, the stabilizers production runs rely on, and which audited tasks reach the stage.

Recipes as sequences

Each row is one documented pipeline, steps in the order the source describes them; hover a step for the note. Preference tuning has moved from the center of the recipe (Llama 3) to a light final touch (Llama 4) or nothing (GLM, DeepSeek, MiMo), while on-policy distillation now closes many recipes.

show older sources

Frameworks by sub-stage

Public code per sub-stage. Most RL frameworks also ship SFT and DPO trainers; on-policy distillation is in NeMo-RL, TRL's GKD trainer and Tinker, and a paper with in-house code at Xiaomi.

What keeps a production RL run stable

The corrections and controls that recur across reports and framework defaults, with the source that pins each. Most address the training–inference numerical mismatch or the staleness that asynchrony introduces.

Which audited benchmark tasks reach post-training

Method hooks inside a fixed verl run and weights-only grading dominate; no audited task exercises preference tuning, on-policy distillation, or trainer–rollout coordination.

Sources on this page

Show sources