One training sample through one training step

One training sample through one training step A sequence diagram generated by Archify. cut 8,192 tokens from the token stream, EOS at boundaries input_ids — just indices into the embedding table an input vector for each position logits over the whole vocabulary at each position compare with labels shifted by one gradients update the weights Prepare the sample Forward Loss and backward Data loader · reads by index, not the whole file · Sequence participant Data loader reads by index, not the whole file Training sample · input_ids · labels · mask · Sequence participant Training sample input_ids · labels · mask Embedding · E[id] → d-dimensional vector · Sequence participant Embedding E[id] → d-dimensional vector Transformer · causal mask + doc boundaries · Sequence participant Transformer causal mask + doc boundaries Cross-entropy · with a per-position loss mask · Sequence participant Cross-entropy with a per-position loss mask Legend main request return message