One training step of RLHF with PPO, and of DPO

One training step of RLHF with PPO, and of DPO A workflow diagram generated by Archify. 01 / RLHF with PPO: four models, online 02 / DPO: two models, offline Prompt x · no reference answer · RLHF with PPO: four models, online Prompt x no reference answer Policy generates y · sampling as in part 9 · RLHF with PPO: four models, online Policy generates y sampling as in part 9 Reward model · scores r(x, y) · RLHF with PPO: four models, online Reward model scores r(x, y) Reference π_ref · per-token KL penalty · RLHF with PPO: four models, online Reference π_ref per-token KL penalty Value model · estimates advantage · RLHF with PPO: four models, online Value model estimates advantage PPO update · clipped objective · RLHF with PPO: four models, online PPO update clipped objective Existing pair · x, y_w, y_l · DPO: two models, offline Existing pair x, y_w, y_l Policy π_θ · log-probs of y_w, y_l · DPO: two models, offline Policy π_θ log-probs of y_w, y_l Reference π_ref · log-probs of y_w, y_l · DPO: two models, offline Reference π_ref log-probs of y_w, y_l DPO loss · −log σ(β · log-ratio gap) · DPO: two models, offline DPO loss −log σ(β · log-ratio gap) Gradient step · like SFT, no sampling · DPO: two models, offline Gradient step like SFT, no sampling regenerate with the new policy Legend policy generating text during training model being updated loss and update step input data for the step frozen model, forward pass only