One pretraining step and where people get it wrong

One pretraining step and where people get it wrong A workflow diagram generated by Archify. 01 / One training step EX / Common misconceptions Micro-batch · cut from token stream · One training step Micro-batch cut from token stream Forward in bf16 · logits at every position · One training step Forward in bf16 logits at every position Cross-entropy · next-token loss · One training step Cross-entropy next-token loss Backward, sync · all-reduce / reduce-scatter · One training step Backward, sync all-reduce / reduce-scatter AdamW step · fp32 master weights · One training step AdamW step fp32 master weights Checkpoint, eval · loss, grad norm, MFU · One training step Checkpoint, eval loss, grad norm, MFU Always 20 tokens/param · only minimizes train cost · Common misconceptions Always 20 tokens/param only minimizes train cost bf16 is renamed fp16 · fp32 exponent range · Common misconceptions bf16 is renamed fp16 fp32 exponent range Kaplan was just wrong · different measurement · Common misconceptions Kaplan was just wrong different measurement FSDP is model parallel · still data parallel · Common misconceptions FSDP is model parallel still data parallel Spikes mean bad data · data + model state · Common misconceptions Spikes mean bad data data + model state Lower loss is better · downstream may not follow · Common misconceptions Lower loss is better downstream may not follow next step, learning rate follows schedule Legend compute step on the GPU weight update data: token stream and micro-batch checkpoint and monitoring common misconception