Document, corpus, sequence, sample, batch: five words that are not synonyms

Document, corpus, sequence, sample, batch: five words that are not synonyms A workflow diagram generated by Archify. 01 / Content units — about what was written 02 / Training units — about what the loss is computed on 03 / Why the two rows above do not line up Document · one web page, one file · Content units — about what was written Document one web page, one file Corpus · a set of documents · Content units — about what was written Corpus a set of documents Curated corpus · filtered, dedup, decon · Content units — about what was written Curated corpus filtered, dedup, decon Tokenized dataset · integer IDs only · Training units — about what the loss is computed on Tokenized dataset integer IDs only Training sequence · fixed length 8,192 · Training units — about what the loss is computed on Training sequence fixed length 8,192 Batch · B × T token positions · Training units — about what the loss is computed on · what the GPU actually gets Batch B × T token positions what the GPU actually gets 50,000-token document · spans many samples · Why the two rows above do not line up 50,000-token document spans many samples Multi-document sample · when the pipeline packs · Why the two rows above do not line up Multi-document sample when the pipeline packs one long document one fixed sequence group by source clean · filter · dedup tokenizer cut by length group samples Legend Agent logic Tool action Context / trace Cloud service External system