From raw data to a training sample Diagram

The top row is the path the data takes

Each stage shrinks the corpus and changes its distribution
No stage is a side step outside the training problem
Input is human-written text, output is a matrix of integers

The bottom row is the dataset builder's decisions

The filter threshold decides how many unique tokens remain
Decontamination decides what benchmarks can still measure
The tokenizer and EOS placement decide a sample's boundaries

What often gets overlooked

The model does not read the Internet; it reads what the pipeline chose
One long document can yield many training samples
One training sample can hold tokens from many documents