Data mixture: what the model reads, and how much Diagram

The mixture is a hyperparameter, not a side effect

Concatenate everything and sample by natural size, and web dominates
Upsampling code to make the model good at coding is a deliberate choice
So is downsampling a large but mediocre source

You can learn the mixture instead of guessing

DoReMi uses a small proxy model to estimate domain weights
The weights found are then used to train the large model
In the paper, this improves downstream results in fewer steps

The mixture need not stay fixed for the whole run

Early phases lean on broad web data for knowledge coverage
Later phases increase code, math and curated text
Weights decide how much gradient the model gets from each kind