Reading a trillion-token dataset without loading it into RAM Diagram

The token stream is a mental model, not a file

Data is spread across many shards for parallel reads and retries
Megatron-LM builds a document index, a sample index and a shuffle index
That lets it fetch sample i without rewriting the dataset

Shuffling at this scale is not random.shuffle

Shuffle at several levels: shard, document, sample index and buffer
The goal is enough randomness without rewriting all the data
The order must be reproducible to resume correctly after a crash

The data loader is a distributed-systems problem

Each rank must get a different slice with balanced load
Throughput must be high enough that GPUs never wait for data
Track how many tokens were consumed so checkpoints are meaningful