Datasets for LLMs: From Raw Data to the Training Samples a Model Actually Reads

Part 4 stopped at a sequence of integers: the tokenizer turns text into input_ids. But it did not answer the question that comes right before that - where does that text come from, and who decides what the model gets to read.

When people talk about Large Language Models, most of the time goes to the Transformer architecture, attention, parameter counts and GPUs. Before any tensor flows through a Transformer, though, a different problem has to be solved first: turning a huge, chaotic and heterogeneous pile of data into token sequences a model can learn from.

A web page is not a training sample. Neither is a book. A repository, a scientific paper, a conversation or a Wikipedia page is just raw data. To become training data, it has to go through collection, content extraction, normalization, quality filtering, deduplication, benchmark decontamination, domain mixing by ratio, tokenization, document-boundary marking and chunking into sequences, and only then is it packed into batches.

The diagram at the top of the post (Figure 1) is that entire journey.

The point worth making is this: building the dataset is not side work that sits outside model training. It is part of the very problem of building an LLM. DataComp-LM showed that the data selection and filtering strategy alone produces very large differences in model quality, even when the training recipe is held relatively fixed. FineWeb, Dolma, RefinedWeb and Nemotron-CC exist for the same reason: the quality, composition and structure of the data directly shape the model you end up with.

This post goes from raw data to exactly what a decoder-only LLM receives during pretraining: a sequence of input_ids, the corresponding target tokens, and the masks needed to compute the loss.

About the code: the snippets in this post are illustrative fragments (token IDs are made up), not runnable scripts.

01What is an LLM’s dataset, really?

In traditional machine learning, a dataset is usually pictured as a labeled table:

image_001.jpg -> cat
image_002.jpg -> dog

Pretraining a causal language model does not start like that. The initial data is just documents containing text:

Document 1:
"The Transformer architecture was introduced in 2017..."

Document 2:
"Python uses indentation to define code blocks..."

Document 3:
"Photosynthesis converts light energy into chemical energy..."

Nobody labels them. The model creates its own supervision from the text itself, through next-token prediction: every token becomes the answer for the tokens that come before it.

That means there is a chain of concepts to keep apart, and they are not synonyms.

1.1 Document

A document is a relatively self-contained unit of content: a web page, a news article, a book chapter, a code file, a forum post, a Wikipedia article.

Documents usually come with metadata:

{
  "text": "Transformers use attention mechanisms...",
  "source": "web",
  "url": "https://example.com/article",
  "language": "en",
  "timestamp": "2026-01-10",
  "quality_score": 0.91
}

Metadata may never go into the model directly, but it is what drives all the filtering, sampling, provenance and auditing downstream. Without it, you cannot answer the question “where did this data come from?”.

1.2 Corpus and curated corpus

A corpus is a collection of many documents, usually still very raw: a Common Crawl snapshot, a Wikipedia dump, a set of GitHub repositories, a collection of public-domain books.

After cleaning, filtering, deduplication and the other quality-control steps, what remains is called a curated corpus.

The single most important distinction in this whole post lives here:

A corpus is about a body of content. A training sample is about how that content is turned into the unit the loss is computed on.

1.3 Tokenized dataset, sequence, sample, batch

The tokenizer turns text into token IDs:

"Language models learn patterns"
        ↓
[6732, 4211, 3876, 12987]

From this point on, the pipeline mostly works with integer token IDs, not strings.

A training sample is a token sequence of a fixed length - 2,048, 4,096 or 8,192 tokens. Several sequences are grouped into a batch. If batch_size = 32 and sequence_length = 8192, one batch holds 32 × 8192 = 262,144 token positions, before you account for parallelism or gradient accumulation.

A document that is 50,000 tokens long produces many training samples. Conversely, one training sample can contain tokens from several documents if the pipeline allows them to be joined. That is why counting documents tells you nothing about training compute.

An LLM does not “read the Internet”. It receives integer matrices representing token sequences that the data pipeline prepared in advance.

02Where does LLM training data come from?

A general-purpose LLM needs to learn many different kinds of patterns, so a pretraining corpus rarely has just one source.

Dolma is a good example of a multi-source corpus: it combines web content, scientific papers, code, public-domain books, social media and encyclopedic material. RefinedWeb asks the opposite, and more interesting, question: if web data is filtered and deduplicated well enough, is the web alone enough to produce a strong model? Their results show that in many settings, the answer is yes.

But “many sources” does not automatically mean “a good dataset”. Each source brings its own problems: the web has spam, SEO pages, boilerplate and duplicated content; code has licenses, secrets and auto-generated files; papers carry noise from PDF extraction; forums contain toxic content and PII; books raise copyright and provenance issues.

So a pipeline usually has two layers:

• source-specific processing - logic tailored to each source;
• source-agnostic processing - shared steps such as deduplication, PII filtering and decontamination.

Dolma describes its pipeline in exactly this way: source code and web data cannot be cleaned identically, even though they end up in the same training mixture.

There is one more distinction that often gets blurred:

LLM data
├── Pretraining data
│   ├── web
│   ├── code
│   ├── books
│   └── ...
│
└── Post-training data
    ├── instruction-response pairs
    ├── preference pairs
    └── synthetic conversations

This post focuses on pretraining data. Instruction tuning and preference learning are different stages, because their sample structure and the way they compute the loss differ considerably.

03From the Internet to a usable corpus

Say we have crawled a web page:

<html>
  <head><title>Understanding Transformers</title></head>
  <body>
    <nav>Home | About | Subscribe</nav>
    <article>
      <h1>Understanding Transformers</h1>
      <p>A Transformer processes tokens using attention...</p>
    </article>
    <aside>Recommended posts...</aside>
    <footer>Copyright 2026...</footer>
  </body>
</html>

We don’t want the model to spend capacity learning Home | About | Subscribe, Accept cookies or Sign up for newsletter. We want to keep the part that has value.

For large-scale web data, Common Crawl is the source most often used in research. It provides three formats worth remembering: WARC holds the raw crawl data and HTTP responses, WAT holds metadata computed from the WARC files, and WET holds pre-extracted plaintext.

3.1 Extracting text is not as simple as stripping HTML tags

A common mental shortcut is to assume that HTML -> strip_tags() -> clean text is enough.

In reality, a web page mixes main content, navigation, cookie banners, comments, ads, related posts, footers, captions, code blocks, tables and formulas. An extractor that is too naive produces something like:

Home About Pricing Log in
Transformer Transformer Transformer
Share Facebook Twitter LinkedIn
Cookie Settings Privacy Policy

On the other hand, an extractor that is too aggressive can delete code blocks, tables or formulas - exactly the parts most worth learning in a technical article.

So extraction is itself a design decision. Datasets built for code or math often need a different pipeline from generic web text: modern math data pipelines even try to preserve MathJax, KaTeX or MathML instead of blindly flattening everything into plain text.

3.2 Normalization cuts both ways

After extraction, text is usually normalized: Unicode normalization, whitespace normalization, encoding fixes, control-character removal, line cleanup, language identification.

Raw:
"LLM  training     is    interesting"

Normalized:
"LLM training is interesting"

But normalization needs care. Stripping too aggressively can break the structure of code, markdown, poetry, tables, or languages that don’t use the Latin alphabet. For Vietnamese, forcing lowercase or stripping accents is the fastest way to ruin the data (Vietnamese marks tones with diacritics, so ma, má, mà and mã are different words). In Python and Markdown, whitespace is real syntax.

3.3 Language ID is metadata, not a filter

If you want an English-only corpus, the pipeline needs to identify the language of each document or each paragraph. If you want a multilingual model, language ID still matters - because it is what lets you control the sampling ratio between languages.

Document A -> English    -> 0.99
Document B -> Vietnamese -> 0.97
Document C -> Mixed      -> uncertain

These results feed straight into the data mixing step later on. A wrong label here becomes a wrong ratio there.

At the end of this stage, you have a set of plain-text documents that is cleaner than the raw crawl - but still very far from a dataset worth spending compute on.

04Having text doesn’t make it worth learning

The Internet contains an enormous amount of text, but most of it is not useful for language modeling:

BUY NOW BUY NOW BUY NOW !!!
DISCOUNT DISCOUNT DISCOUNT
CLICK HERE CLICK HERE CLICK HERE

or an auto-generated page:

Cheap Hotel Paris
Cheap Hotel London
Cheap Hotel Bangkok
Cheap Hotel Tokyo
...

Put into the corpus as-is, these patterns eat into the token budget and skew the very distribution the model learns.

4.1 Heuristics: cheap enough to run at any scale

You can measure very simple features: document length, the ratio of alphabetic characters, the ratio of punctuation, average word length, the ratio of repeated lines, the ratio of symbols and digits, the number of boilerplate phrases, paragraphs without punctuation, or unwanted URL patterns and file extensions.

if alphabetic_ratio < threshold:
    discard(document)

if repeated_line_ratio > threshold:
    discard(document)

The big advantage: they are cheap and run at the scale of trillions of tokens. The downside is that they easily introduce bias. A rule designed for English prose will inadvertently knock out code, poetry, tables, or text in another language.

4.2 Model-based filtering

Instead of relying only on rules, you train a classifier to predict document quality, learning from labels assigned by humans or by another model.

A prominent example is FineWeb-Edu. The authors built a classifier to find text with educational value in FineWeb; according to the dataset card and the paper, filtering aggressively to keep a subset of about 1.3T educational tokens significantly improved many knowledge and reasoning benchmarks compared with training on less selective web data.

DataComp-LM reached a similar conclusion: model-based filtering is a very effective component in building a high-quality training set. In that benchmark, a good curation strategy produced a significantly better model without simply adding compute.

4.3 The threshold is a precision-recall problem

Suppose the Internet has 100 documents, 30 of them good and 70 poor.

A filter that is too loose keeps 80 documents: 30 good, 50 poor. A filter that is too strict keeps 12 documents: 12 good, 0 poor.

The second filter has a higher average quality, but you lose 18 good documents. At small scale this doesn’t matter much. Over a training horizon of trillions of tokens, it leads to a shortage of unique data and forces the model to read the same data several times.

Nemotron-CC focuses precisely on this trade-off: rather than aggressively throwing away most of the web, they combine quality classifiers, synthetic rephrasing and a few other strategies to keep a larger volume of data for long-horizon pretraining.

There is no “right” threshold for every model. The right threshold depends on the model size, the token budget, the target capabilities and the length of the training horizon.

05Why a model can’t read the same thing a thousand times

The web is a giant copying machine. One article can show up on its original site, an RSS mirror, a news aggregator, a scraped blog, a cached page, a forum that quotes it and an SEO clone.

Doc A: "Attention is all you need..."
Doc B: "Attention is all you need..."
Doc C: "Attention is all you need..."
Doc D: "Attention is all you need, and the Transformer..."

There are three levels of duplication to think about: exact duplicates (identical), near duplicates (almost identical), and repeated substrings (a long passage repeated inside otherwise different documents).

5.1 Duplication skews the distribution, not just the storage bill

Suppose the true distribution is one document per topic. But Topic A has been copied 100 times across the Internet:

Observed corpus
Topic A: 100 copies
Topic B:   1 copy
Topic C:   1 copy

The model gets far more gradient from Topic A, even though that repetition is just an artifact of the web crawl and says nothing about importance.

Deduplicating Training Data Makes Language Models Better by Lee et al. showed that language datasets contain a substantial amount of near-duplicate content; models trained on deduplicated data emit less memorized text and can reach the same or better quality in fewer training steps.

5.2 Exact hashing misses near duplicates

The simplest approach is to normalize, hash the content, and drop the document if that hash has been seen before. But:

A: Transformers are powerful language models.
B: Transformers are powerful language models!

One character apart, and the hashes are completely different. Exact hashing is still cheap and useful: Dolma’s web pipeline runs exact deduplication by URL, by document and by paragraph (with a Bloom filter), and each step removes a substantial share of the data. But any lightly edited copy - a changed punctuation mark, an inserted timestamp, a different template - slips through. That is the near-duplicate zone, and it needs a different tool.

5.3 Near duplicates: shingles, MinHash, LSH

For near duplicates, you represent each document as a set of n-gram shingles. For example, with 3-grams:

"large language model training"

large language model
language model training

Then you measure the overlap between two shingle sets with Jaccard similarity - the size of the intersection divided by the size of the union.

But computing Jaccard for every pair of documents is close to quadratic in the number of documents, which is infeasible with billions of documents. So large pipelines use an approximation: MinHash compresses each document into a short signature, LSH puts similar signatures into the same bucket, and you only check carefully within each bucket.

Modern corpora also deduplicate at several levels: by URL, by document, by paragraph, or globally across multiple crawl snapshots.

5.4 More aggressive is not always better

An interesting result from FineWeb: running MinHash deduplication globally across all crawls improved benchmarks only slightly, well behind deduplicating each crawl separately. Comparing what was removed with what was kept in an older crawl, the authors found the retained data was of lower quality: global deduplication had inadvertently upsampled low-quality data. Their hypothesis is that the main gain from deduplication comes from removing the very large duplicate clusters present in every crawl, while further deduplicating small clusters (fewer than about 100 copies, roughly the number of crawls) can hurt performance.

Duplication is noise, but it also acts, unintentionally, as a sampling weight. Deduplication is therefore also a decision about the distribution of the training data.

06Contamination: when the model has seen the exam

Deduplication handles duplication inside the corpus. Decontamination handles a different problem: data used for evaluation ends up inside the data used for training.

Suppose you use MMLU-style questions to score a model:

Question: ...
A. ...
B. ...
C. ...
D. ...
Answer: C

If some page on the Internet has published the exact question together with its answer, a web crawl can easily pick it up.

In that case, the model can score highly not because it generalizes better, but because it has already seen nearly identical content.

There are four levels to tell apart:

• exact contamination - the benchmark example verbatim;
• near-duplicate contamination - the question reformatted or with a few words changed;
• semantic contamination - the same content and answer logic, phrased differently;
• benchmark discussion contamination - solutions, answer keys or write-ups of the questions appearing online.

Decontamination can use exact matching, n-gram overlap, substring matching, hashing, Bloom filters or near-duplicate detection. Dolma, for example, does it by finding paragraphs of the evaluation datasets that appear in training documents. The goal is not to prove contamination is absolutely zero, but to reduce the obvious cases that inflate scores.

The problem gets harder when a model’s training corpus is not public - from the outside, it is very hard to know whether the model has seen a benchmark. This is one of the reasons modern LLM evaluation needs many new benchmarks, dynamic evaluation and contamination analysis.

A simple way to think about it:

Benchmark score
      =
reasoning ability and knowledge
      +
memorization of what it may have seen
      +
measurement noise

Without controlling contamination, you cannot separate those three parts from each other.

07Data mixture: what the model should read, and how much

Suppose that after cleaning you have: web 8 TB, code 1 TB, papers 300 GB, books 200 GB, Wikipedia 100 GB.

The naive option is to concatenate everything and sample by natural size. Then the web dominates almost completely:

Natural distribution

Web        █████████████████████████████████
Code       ████
Papers     █
Books      █
Wikipedia  ▌

But that is not necessarily the best distribution for the model. You can deliberately build a training mixture:

Training mixture

Web        ██████████████████████
Code       ████████
Papers     ████
Books      ████
Wikipedia  ██

7.1 Sampling weights

Suppose the domains have weights w_web = 0.55, w_code = 0.20, w_books = 0.10, w_papers = 0.10, w_wiki = 0.05, summing to 1. Every time you draw data, you sample a domain from this distribution. These numbers are only illustrative, not a standard recipe.

If code makes up only about 10% of the raw corpus (1 TB out of 9.6 TB in the example above) but you want the model to be stronger at coding, you can upsample code. If a web source is very large but of average quality, you can downsample web.

The data mixture decides how much gradient the model gets from each kind of content. In other words, it is a pretraining hyperparameter, not a natural side effect of collecting data.

7.2 DoReMi: learning the mixture instead of picking it by intuition

DoReMi is an important example. Instead of choosing domain weights purely by feel, the method uses a small proxy model to estimate a better mixture, and only then uses those weights to train the large model. In the paper, optimizing the mixture improved average downstream performance and reached baseline quality in fewer training steps in the specific experiments reported.

7.3 The mixture can change over time

Nothing requires the whole training run to use the same distribution. A simple curriculum: lean toward broad web data early on to cover general knowledge, then increase the share of code, math and carefully curated text later.

08How big is a dataset? What do you count?

A dataset can be described by three numbers: 500 GB, 100 million documents, 2 trillion tokens. These three numbers are not equivalent.

8.1 Neither bytes nor documents are enough

Two corpora of 1 TB each can have very different token counts if they differ in encoding, language or tokenizer, contain a lot of code, JSON or markup, or compress differently. And documents range from one that is 20 tokens long to another that is 100,000 tokens long.

So in language-model training, the quantity that tracks compute most closely is the number of training tokens.

8.2 Model size, tokens and compute are tied together

Scaling laws don’t simply say that more parameters is better. You have to look at the parameter count, the number of training tokens and the total compute together.

The Chinchilla study is an important milestone: DeepMind showed that many earlier models were undertrained, in the sense of having too many parameters for the number of tokens they were trained on under the same compute budget. Chinchilla has 70B parameters but was trained on about 1.4T tokens, and it beat the 280B Gopher on many evaluations while using a comparable amount of compute.

The lesson is not “every model must follow one fixed tokens-per-parameter ratio”, because the optimal recipe depends on many factors. The more important lesson is:

If you increase model size without increasing the amount of training data to match, the model may not make full use of its own capacity.

A few markers for scale: Meta said Llama 3 was pretrained on more than 15T tokens; FineWeb introduced a web corpus of about 15T tokens (later updates have grown it to more than 18.5T); Dolma released a corpus of about 3T tokens.

09From documents to token sequences

At this point you have a curated corpus:

Document A: "The cat is sleeping."
Document B: "Transformers use attention."
Document C: "Python is a programming language."

The tokenizer turns each document into IDs, and the pipeline inserts special tokens to preserve the boundaries:

Doc A -> [314, 982, 71, 402, 13, EOS]
Doc B -> [812, 119, 734, 13, EOS]
Doc C -> [501, 27, 84, 913, 42, 13, EOS]

After that, you can think of them as one logical token stream:

[314, 982, 71, 402, 13, EOS,
 812, 119, 734, 13, EOS,
 501, 27, 84, 913, 42, 13, EOS,
 ...]

This is the direct link between the Tokenization post and this one. The tokenizer decides text -> token IDs. The data pipeline decides token IDs -> training sequences.

9.1 EOS is more than a period

EOS - End-of-Sequence or End-of-Document - tells the model that a unit of content has ended. It has two different roles depending on the pipeline:

1. it becomes a target that the model learns to predict;
2. it marks a boundary so attention or position IDs can be reset or masked if the recipe calls for it.

Not every implementation uses the boundary in the same way. That is why you need to separate a token boundary from an attention boundary - having an EOS does not automatically mean attention is blocked there.

9.2 The token stream does not exist as one giant file

The “concatenate all the tokens” picture is mainly a mental model. In a production-scale pipeline, data is stored in shards, with an index that maps each training sample to a document id, a start offset and an end offset.

Megatron-LM, for example, builds mappings such as a document index, a sample index and a shuffle index to fetch samples efficiently from an indexed dataset, instead of physically creating a single token-stream file for everything.

10Chunking, document boundaries and so-called “packing”

Suppose the model is trained with sequence_length = 8 and the logical token stream is:

[12, 81, 53, 9, EOS, 72, 19, 44, 68, EOS, 101, 91, ...]

You cut it into:

Sample 1: [12, 81, 53, 9, EOS, 72, 19, 44]
Sample 2: [68, EOS, 101, 91, ...]

10.1 Why not pad each document

If every document is shorter than the sequence length and you pad each one separately:

Doc A: [A A A EOS PAD PAD PAD PAD]
Doc B: [B B EOS PAD PAD PAD PAD PAD]

The GPU still performs computation on the PAD positions, but those positions contribute no content. Joining several documents into the same fixed-length sequence gives much higher utilization:

[A A A EOS B B EOS C]

A note on terminology: in pretraining, people usually talk about concatenating documents and chunking the token stream. In supervised fine-tuning, “sequence packing” usually refers to the technique of joining several short training examples into one sequence to reduce padding waste, while keeping metadata and masks so the examples don’t affect each other. The two ideas are very close in their goal of compute utilization, but you shouldn’t assume the implementations are identical.

10.2 Two options when a sequence holds several documents

If Sample 1 contains Doc A: [A1 A2 A3 EOS] followed by Doc B: [B1 B2 B3 B4], there are two directions.

Let attention cross the boundary. B1 can attend to A1 A2 A3 EOS. The model receives unrelated context between the two documents, but the implementation is simple and makes full use of the continuous token stream.

Mask attention across document boundaries. Conceptually, the attention matrix becomes two separate blocks:

      A1 A2 A3 E  B1 B2 B3 B4
A1    x
A2    x  x
A3    x  x  x
E     x  x  x  x
B1                x
B2                x  x
B3                x  x  x
B4                x  x  x  x

Llama 3 is a public example: Meta said the model was trained on sequences of 8,192 tokens and used a mask so that self-attention does not cross document boundaries.

The point to remember, in both directions:

A fixed-length training sequence does not necessarily correspond to one document. And one long document can be split across several training sequences.

11What does a causal LLM training sample look like?

This is the last step before the data goes into the Transformer. Suppose the token sequence is:

[BOS, I, love, machine, learning, EOS]

The goal of causal language modeling is to learn the probability of the whole sequence as a product of conditional probabilities: each token is predicted from the tokens before it. Conceptually:

Input:   [BOS, I,    love,    machine, learning]
Target:  [I,   love, machine, learning, EOS]

11.1 input_ids

After the tokenizer:

input_ids = [1, 40, 912, 7310, 4481, 2]

The model does not see the words I, love, machine. It sees integer IDs and uses the embedding table in the first layer to turn them into vectors.

11.2 labels and the shift

There are two common ways of putting it, and you need to separate the concept from the implementation. Conceptually, input = x[0 : T-1] and target = x[1 : T]. In many implementations, labels is passed in almost identical to input_ids, and the model or the loss function shifts the logits and labels internally.

So when you see:

labels = input_ids

it does not mean the model learns to “predict the current token itself”. The one-step shift still happens when the next-token loss is computed.

11.3 Three kinds of mask in the same sample

Besides document-boundary masking, a decoder-only Transformer also needs causal masking - the token at position t may only look at positions less than or equal to t:

      t1 t2 t3 t4
 t1   x
 t2   x  x
 t3   x  x  x
 t4   x  x  x  x

If the token at position 2 were allowed to see the token at position 3 during training, the next-token prediction task would leak.

The third kind is the loss mask: not every position has to contribute to the loss. For [real real real PAD PAD] you use the mask [1, 1, 1, 0, 0]. Likewise, a recipe can decide to drop the loss on certain special tokens or boundary tokens.

A real training sample can therefore be described conceptually as:

{
  "input_ids":      [12, 81, 53, 9, 72, 19, 44, 68],
  "labels":         [12, 81, 53, 9, 72, 19, 44, 68],
  "attention_mask": "causal + optional document boundaries",
  "loss_mask":      [1, 1, 1, 1, 1, 1, 1, 1]
}

The exact format varies by framework. But the essence does not change: text goes into the tokenizer, comes out as input_ids, becomes vectors through the embedding, becomes logits through the Transformer, and cross-entropy compares the logits with the next-token targets.

12Reading a trillion-token dataset

A dataset of a few GB can be loaded fairly simply. A few trillion tokens cannot. You can’t write dataset = load_everything_into_ram() and expect it to work.

12.1 Sharding and streaming

The dataset is split into many smaller files:

dataset/
├── shard_00000.bin
├── shard_00001.bin
├── ...
└── shard_99999.bin

The benefits: easy distribution on object storage, many workers reading in parallel, easy retries on failure, easy shuffling by shard, and no single giant file. Data is read sequentially from disk or remote storage instead of loading everything into RAM.

With preprocessed token IDs, a memory-mapped binary file lets a process access the region of data it needs as if it were one large array, without reading the whole file into memory. Megatron-LM supports indexed datasets, mmap and the mappings that determine sample boundaries - a textbook example of how the data pipeline also has to scale with the training system.

12.2 Shuffling at this scale is not random.shuffle

You still want randomness, but you can’t call random.shuffle(all_examples) on trillions of tokens. Systems shuffle at several levels: shuffling documents, shuffling shards, shuffling sample indices, shuffling within a buffer.

A mapping might look like:

sample 0 -> document 918, offset 1200
sample 1 -> document  41, offset  320
sample 2 -> document 551, offset 7700

That way training gets a random sample order without rewriting the entire dataset.

12.3 Distributed data loading

With thousands of GPUs, each worker needs to receive the right slice of data and avoid all GPUs training on the same sample at the same time. The pipeline has to guarantee: reproducibility, deterministic resume after a crash, balanced load, enough throughput that GPUs don’t wait for data, and tracking of how many tokens have been consumed.

This is where we start to cross into training an LLM from scratch, a topic planned for a later part of the series, because the data loader, distributed training, the optimizer and checkpointing interact closely. The thing to remember is that at LLM scale, data is no longer a CSV loaded into RAM but a distributed-systems problem.

13Train, validation, benchmarks and leakage

In ordinary supervised learning you split a dataset into train, validation and test. In LLM pretraining the idea survives but gets more complicated.

The training corpus is the part the model learns gradients from directly. The validation corpus is used to track loss or perplexity on held-out data and to detect overfitting or distribution mismatch. Benchmarks such as MMLU, ARC or HumanEval are usually external evaluation sets, not a random 10% split of the pretraining corpus - and, importantly, they need to be as independent of the training data as possible.

13.1 A random split can still leak

Suppose Doc A is the original article and Doc B is a copy. If a random split puts Doc A in train and Doc B in validation, the validation set is not truly independent.

So deduplication needs to happen before splitting, or at least be checked across splits.

Some projects also use a temporal split or a source-based split where appropriate:

Train:      documents before date X
Validation: documents after date X

This measures generalization over time better, but it also changes the distribution and is not always appropriate.

14Privacy, copyright, bias and provenance

A data pipeline does not only optimize benchmark scores. A web-scale corpus can contain personal data, emails, phone numbers, copyrighted works, toxic content, hate speech, dangerous instructions, secrets and API keys, along with very skewed cultural representation.

14.1 PII and secrets

PII - Personally Identifiable Information - can include emails, phone numbers, IP addresses and many other kinds of identifiers. A pipeline can detect -> mask or detect -> drop document. Dolma describes using carefully crafted regular expressions (rather than model-based detectors, which are too slow at its scale) to detect and mask some PII, such as email addresses, phone numbers and IP addresses, in its pipeline.

Public repositories can also accidentally contain:

API_KEY=...
PRIVATE_TOKEN=...
PASSWORD=...

Code datasets therefore need secret scanning and license-aware filtering, not just text cleaning.

14.2 Copyright and licenses

“Publicly accessible” does not mean “not copyrighted”. The questions you need to be able to answer: where the data came from, what license it carries, whether it may be redistributed, whether there is an opt-out mechanism, and whether there is a provenance record. This is why metadata keeps getting more important in dataset engineering.

14.3 Bias comes from both the data and the filters

A model learns the distribution of its data. If 90% of a so-called multilingual corpus is actually English, the model will not be equally capable across languages. If an occupation, a country, a gender or a community is represented in a skewed way in the data, the model can absorb exactly that pattern.

More notably: filtering also creates new bias. A “quality” classifier trained on encyclopedic prose can penalize everyday language, dialects, or content from underrepresented communities.

Data quality is not an entirely objective concept. It depends on the model’s goals and on the criteria of whoever builds the dataset.

15Synthetic data: a dataset isn’t only written by people

An important trend in modern LLMs is using models to create or transform the training data itself. Synthetic data is especially common in post-training, but it increasingly shows up in pretraining and continued pretraining as well. Nemotron-CC is a clear example: the corpus combines real web tokens with synthetic rephrasing to increase the amount of high-quality data usable for long-horizon training.

Suppose the raw web text is:

"Photosynthesis is the process plants use to make food."

A teacher model can produce a cleaner, more informative version:

"Photosynthesis is a biological process in which plants convert light
energy into chemical energy, primarily in chloroplasts."

Synthetic data can help standardize writing quality, create more reasoning data, add more code and math examples, supplement low-resource domains, rewrite noisy web content, and build a curriculum with controllable difficulty.

15.1 But it is not a free lunch

If the teacher is wrong, the synthetic sample is wrong, and the student learns exactly that mistake. If the teacher is biased, the generated distribution carries that bias, and the student inherits it. If the whole ecosystem trains over and over on model-generated text without quality control, the distribution can narrow and lose diversity.

So a synthetic pipeline usually needs all five steps, not just the first:

generation
   ↓
quality scoring
   ↓
verification
   ↓
deduplication
   ↓
sampling

15.2 The data loop

There is an increasingly clear loop: better models yield better quality classifiers and teachers that generate better data; better data in turn produces better models in the next round. The verification step is what keeps that loop from poisoning itself.

16Common misconceptions

The dataset is the part of a model’s behavioral architecture that isn’t in the weights file

When you look at an LLM, you see billions of parameters. But every parameter is updated by gradients, and the gradients come from the tokens the data pipeline chose to show the model. The real causal chain is: data sources, then curation decisions, then the training distribution, then gradients, and only then the model’s capabilities, biases and memorization behavior.

That makes the dataset part of the model’s “behavioral architecture”, even though it is not in the weights file once training is over.

What to take with you:

• a corpus is about content, a training sample is about the unit the loss is computed on - two different things;
• quality filtering is a precision-recall problem, and there is no threshold that is right for every model;
• deduplication changes the distribution, it doesn’t just save space;
• decontamination is the precondition for a benchmark to still mean something;
• the data mixture is a pretraining hyperparameter, and it can be learned instead of guessed;
• the token count, not GB or the number of documents, is the quantity closest to compute;
• a fixed-length sequence does not correspond to one document, in either direction;
• every filter carries the bias of whoever designed it.

And in the end, what the model receives is no longer:

"The Transformer architecture uses self-attention..."

but something closer to:

input_ids = [8812, 13, 7021, 421, 993, 511, ...]
labels    = [8812, 13, 7021, 421, 993, 511, ...]

together with a causal mask, document-boundary information and an appropriate loss mask. That is the handoff point from data engineering to model computation.

The next step goes one stage further: how those integer token IDs are turned into vectors, and why the distance between two vectors can carry meaning. That is where embedding begins - the subject of the next part of the series (Vietnamese only for now).

References

• Stanford CS336 — Language Modeling from Scratch
• Li et al. — DataComp-LM: In search of the next generation of training sets for language models
• Penedo et al. — The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
• Soldaini et al. — Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research
• Penedo et al. — The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only
• Lee et al. — Deduplicating Training Data Makes Language Models Better
• Xie et al. — DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining
• Hoffmann et al. — Training Compute-Optimal Large Language Models
• Su et al. — Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Dataset
• Meta — Introducing Meta Llama 3
• Common Crawl — Get Started: WARC, WAT, WET formats
• NVIDIA — Megatron-LM GPT dataset and indexed data loading