Post-training from base model to assistant, and where people get it wrong

Post-training from base model to assistant, and where people get it wrong A workflow diagram generated by Archify. 01 / The RLHF branch 02 / Main path (DPO branch) EX / Where people get it wrong Reward model · Bradley–Terry · The RLHF branch Reward model Bradley–Terry PPO · reward − β·KL to reference · The RLHF branch PPO reward − β·KL to reference Base model · continues text (part 10) · Main path (DPO branch) Base model continues text (part 10) SFT · chat template, loss mask · Main path (DPO branch) SFT chat template, loss mask Preference pairs · human or AI raters · Main path (DPO branch) Preference pairs human or AI raters DPO · loss directly on pairs · Main path (DPO branch) DPO loss directly on pairs Assistant · repeated over rounds · Main path (DPO branch) Assistant repeated over rounds Teaches new facts · mostly teaches behavior · Where people get it wrong Teaches new facts mostly teaches behavior Loss on prompt too · assistant tokens only · Where people get it wrong Loss on prompt too assistant tokens only CAI has no humans · humans write principles · Where people get it wrong CAI has no humans humans write principles DPO always wins · still debated · Where people get it wrong DPO always wins still debated DPO drops reference · π_ref is in the loss · Where people get it wrong DPO drops reference π_ref is in the loss Reward = quality · reward hacking · Where people get it wrong Reward = quality reward hacking Legend training step input and output model data: conversations, comparison pairs misconception