One RLVR step with GRPO, and the DeepSeek-R1 pipeline

One RLVR step with GRPO, and the DeepSeek-R1 pipeline A workflow diagram generated by Archify. 01 / One GRPO step (main) 02 / Constraints and weak spots 03 / DeepSeek-R1 pipeline Prompt · math, code with answers · One GRPO step (main) Prompt math, code with answers Policy samples G · sampling, T > 0 · One GRPO step (main) Policy samples G sampling, T > 0 Verifier scores · right/wrong + format · One GRPO step (main) Verifier scores right/wrong + format Group advantage · (r − mean) / std · One GRPO step (main) Group advantage (r − mean) / std Update policy · clip, minus β·KL · One GRPO step (main) Update policy clip, minus β·KL Leaky verifier · edit tests, skip tests · Constraints and weak spots Leaky verifier edit tests, skip tests Uniform group · all right/wrong: A = 0 · Constraints and weak spots Uniform group all right/wrong: A = 0 Reference model · frozen · Constraints and weak spots Reference model frozen DeepSeek-V3-Base · base model · DeepSeek-R1 pipeline DeepSeek-V3-Base base model Cold-start SFT · thousands of long CoTs · DeepSeek-R1 pipeline Cold-start SFT thousands of long CoTs Reasoning RL · accuracy + language · DeepSeek-R1 pipeline Reasoning RL accuracy + language Rejection sampling · + SFT on ~800k samples · DeepSeek-R1 pipeline Rejection sampling + SFT on ~800k samples RL, all scenarios · adds a reward model · DeepSeek-R1 pipeline RL, all scenarios adds a reward model KL next step Legend R1 training stage policy samples and updates scoring and advantage frozen reference model where signal is lost or exploited